[![Mergify Status][mergify-status]][mergify]
The idea of using an S3 bucket as YUM repository is not new. Several blog posts are available on the Internet.
We will use the AWS Cloud Development Kit (CDK) to setup the S3 bucket. By doing so we are following the Infrastructure-as-Code (IaC) pattern.
To keep it simple, the repository for CentOS 8 is setup locally and then synced to the bucket using the AWS CLI.
Install the createrepo package and tools required at later stage
yum -y install createrepo wgetCreate directory s3_yum_repository. It will hold the repository content
mkdir s3_yum_repositoryBy executing createrepo ./s3_yum_repository the directory is scanned and metadata
files generated.
$ createrepo ./s3_yum_repository
Directory walk started
Directory walk done - 0 packages
Temporary output repo path: ./s3_yum_repository/.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finishedThe metadata is stored in the repodata sub-directory
$ ls -1 s3_yum_repository/repodata
50da989dd2394a07472c95ad16876baef03e8b936a7ff20e8df393e51cc85c02-other.sqlite.bz2
54cde142e5b5449d2d94fac6b94b8ec96d0a18c813fcdd31b318bddded1d1f74-filelists.xml.gz
a7bb37a6ae126c74b873e11d90d28988a84be7d21f4fe0c5ba00fedad06bf54d-primary.xml.gz
abc202b95b85fdd9cf79dd4a2544b0d4d774e97bce42a41847488cfb45fbcb05-other.xml.gz
bd2ec9eb92cf7949610c0ed9dd52c065ccbe1ca02e704f29a7b0629bec6d3add-filelists.sqlite.bz2
f26b1897a3385e13b7376de0e1c87663dc2ddcd4532dd27004ac99d96c5e3d43-primary.sqlite.bz2
repomd.xmlPrepare the sub-directory structure
mkdir -p s3_yum_repository/{noarch,x86_64,SRPMS}Only the x86_64 folder will be used for storing our RPMs. Download the files of Adopt JDK & JRE as well as Corretto 11 JDK for this architecture.
wget https://adoptopenjdk.jfrog.io/adoptopenjdk/rpm/centos/8/x86_64/Packages/adoptopenjdk-13-hotspot-13+33-1.x86_64.rpm
wget https://adoptopenjdk.jfrog.io/adoptopenjdk/rpm/centos/8/x86_64/Packages/adoptopenjdk-13-hotspot-jre-13+33-1.x86_64.rpm
wget https://d3pxv6yz143wms.cloudfront.net/11.0.3.7.1/java-11-amazon-corretto-devel-11.0.3.7-1.x86_64.rpmMove the files to the repository directory
mv ./adoptopenjdk-13-hotspot-13+33-1.x86_64.rpm s3_yum_repository/x86_64
mv ./adoptopenjdk-13-hotspot-jre-13+33-1.x86_64.rpm s3_yum_repository/x86_64
mv ./java-11-amazon-corretto-devel-11.0.3.7-1.x86_64.rpm s3_yum_repository/x86_64Re-create the repository data
$ createrepo --update ./s3_yum_repository
Spawning worker 0 with 1 pkgs
Spawning worker 1 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs completeThe bucket will not contain any kind of sensitive data, so the encryption is turned off.
To restrict the access to the bucket the blockPublicAccess and publicReadAccess are explicitly listed.
const bucket: Bucket = new Bucket(this, "CorrettoS3Bucket", {
blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
encryption: BucketEncryption.UNENCRYPTED,
publicReadAccess: false,
removalPolicy: RemovalPolicy.DESTROY,
versioned: false,
});Two policy statements are set up to allow access to the bucket content from a specific IP address (e.g. the one I got from my ISP). The first one defines who is allowed to read objects (in our case e.g. the RPM files)
const bucketContentStatement = new PolicyStatement({
effect: Effect.ALLOW,
actions: ["s3:GetObject"],
resources: [bucket.bucketArn + "/*"],
principals: [new AnyPrincipal()],
conditions: {
IpAddress: {
"aws:SourceIp": ["87.122.210.145/32"],
},
},
});The second defines who is allowed to list the contents of the bucket
const bucketStatement: PolicyStatement = new PolicyStatement({
effect: Effect.ALLOW,
actions: ["s3:ListBucket", "s3:GetBucketLocation"],
resources: [bucket.bucketArn],
principals: [new AnyPrincipal()],
conditions: {
IpAddress: {
"aws:SourceIp": ["87.122.210.145/32"],
},
},
});Both statements are added to the bucket policy
const bucketPolicy = new BucketPolicy(this, "bucketPolicy", {
bucket: bucket,
});
bucketPolicy.document.addStatements(
bucketContentStatement, bucketStatement);Synchronize the content of your local YUM repository to the S3 bucket
aws s3 sync --profile cdk s3_yum_repository s3://correttoyumrepositorysta-correttos3bucketbbeb0a25-1p908wuzpckvj For the test a CentOS 8 virtual machine can be setup by
- Using the ISO image
CentOS-8.1.1911-x86_64-boot.isoas starting point - Choose minimal setup to define the set of packages to install
Alternatively a Docker container can be used - this approach is the preferred one. Downloading the Docker image to localhost is a one-liner
$ docker pull centos:8
8: Pulling from library/centos
8a29a15cefae: Pull complete
Digest: sha256:fe8d824220415eed5477b63addf40fb06c3b049404242b31982106ac204f6700
Status: Downloaded newer image for centos:8
docker.io/library/centos:8Spinning up the container from the downloaded image and mounting the directory containing my AWS config and credentials
docker run --name centos_8 -it -v /home/stefan/.aws:/root/.aws --rm centos:8yum -y install python3-pip vim
pip3 install awsclicd /etc/yum.repos.d/
touch s3.repoThe content of the file s3.repo is as below - only the baseUrl needs to be updated based
on the ARN of the S3 bucket.
[s3_repo-x86_64]
name = S3 repository
baseurl = https://correttoyumrepositorysta-correttos3bucketbbeb0a25-1gmfxywyksoie.s3.eu-central-1.amazonaws.com/
gpgcheck = 0
enabled = 1
Installing the Corretto JDK is then possible by running yum
$ yum install java-11-amazon-corretto-devel.x86_64
S3 repository 16 kB/s | 2.9 kB 00:00
[...]
Vorbereitung läuft : 1/1
Installieren : java-11-amazon-corretto-devel-1:11.0.3.7-1.x86_64 1/1
Ausgeführtes Scriptlet: java-11-amazon-corretto-devel-1:11.0.3.7-1.x86_64 1/1
Überprüfung läuft : java-11-amazon-corretto-devel-1:11.0.3.7-1.x86_64 1/1
Installiert:
java-11-amazon-corretto-devel-1:11.0.3.7-1.x86_64
Fertig.S3 treats "+" characters in the path as though they were space characters. Hence renaming of the rpms files is required before
- running
createrepo - syncing the local content to S3