This Terraform Module installs and configures
aws-for-fluent-bit on an EKS cluster, so that
each node runs fluent-bit to collect the logs and ship to CloudWatch Logs, Kinesis Streams, or
Kinesis Firehose.
This module uses the community helm chart, with a set of best practices inputs.
How does this work?
This module solves the problem of unifying the log streams in your Kubernetes cluster to be shipped to an aggregation
service on AWS (CloudWatch Logs, Kinesis, or Firehose) so that you have a single interface to search and monitor your
logs. To achieve this, the module installs a service (fluent-bit) that monitors the log files on the filesystem,
parses custom log formats into a unified format, and ships the result to a centralized log aggregation service
(CloudWatch).
fluent-bit is installed as a Kubernetes
DaemonSet, which ensures that there is one
fluent-bitPod running per node. In this way, we are able to ensure that all workers in the cluster are running the
fluent-bit service for shipping the logs into CloudWatch.
fluent-bit is an optimized version of fluentd that focuses on
streaming and aggregating log files. fluentd has a larger ecosystem of plugins that enable various processing
capabilities on top of the logs prior to aggregating in the data store.
For most EKS deployments, it is recommended to use this fluent-bit module for container log aggregation. Unless you have a specific
need for a plugin only supported by fluentd, the superior performance and memory footprint of fluent-bit will
ensure resources are available on your EKS workers for your Pods.
Log format
This module leverages native plugins for Kubernetes built into fluent-bit that extract additional
metadata for each Pod that is reporting. Each log is shipped to the respective outputs in the following structure:
This allows you to filter and search the logs by the respective attributes. For example, the following CloudWatch
Insights Query can be used to search for all logs from Pods in the kube-system Namespace:
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"2244b0e20ffd429cfa5c9051e3891723dc36d5fb"}]},{"name":".gitignore","path":".gitignore","sha":"7f6cf4bc746bbfd6da4c7a21dbcf1a2296aa0c10"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"b008949ef10a7bad93ab93e8821da77577a30c5c"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"ecbeaab263c59e955b621268f161059633041e3d"},{"name":"CONTRIBUTING.md","path":"CONTRIBUTING.md","sha":"a7cc7bd94443c252390564fa988755dbbe80d87d"},{"name":"GRUNTWORK_PHILOSOPHY.md","path":"GRUNTWORK_PHILOSOPHY.md","sha":"02d9873a74c99fe6d9b6b26bd9f8eb4a7a699c32"},{"name":"LICENSE.md","path":"LICENSE.md","sha":"a2cf01ecdd725fddd718ab91c80c115882c94f3c"},{"name":"README.adoc","path":"README.adoc","sha":"832d4d04377add6a4a946ac90d0c2541fbd6f1d0"},{"name":"_docs","children":[{"name":"eks-architecture.png","path":"_docs/eks-architecture.png","sha":"b4c9c46f88ed465c5575e915af54ad9920b56941"},{"name":"eks-icon.png","path":"_docs/eks-icon.png","sha":"83a29dc46e7bc6234ba5bb825e8ae283c56229a0"},{"name":"iam-role-icon.png","path":"_docs/iam-role-icon.png","sha":"c05bb05e6caae9b9db46505ce505a386f21fa2e4"}]},{"name":"core-concepts.md","path":"core-concepts.md","sha":"ce87f83b7a02e7a5568f46e1b92fe5a5ecc4036b"},{"name":"examples","children":[{"name":"README.md","path":"examples/README.md","sha":"a70f3adc0c888e07b0b03cb32fbd156547c354da"},{"name":"eks-cluster-managed-workers","children":[{"name":"README.md","path":"examples/eks-cluster-managed-workers/README.md","sha":"b8f9642b8db4b69f3e35faee0801f0421f92a83d"},{"name":"dependencies.tf","path":"examples/eks-cluster-managed-workers/dependencies.tf","sha":"c51d22849120296cb44e2637625fbe0ef4405a53"},{"name":"main.tf","path":"examples/eks-cluster-managed-workers/main.tf","sha":"3374927752fdf542c4a63860b5ef3ca78f1b95ff"},{"name":"outputs.tf","path":"examples/eks-cluster-managed-workers/outputs.tf","sha":"431bebd71e3f9d5c299c1740ba16b2eef717cbf0"},{"name":"variables.tf","path":"examples/eks-cluster-managed-workers/variables.tf","sha":"42c1a3ff93506806f637bdb33307341718824136"}]},{"name":"eks-cluster-with-iam-role-mappings","children":[{"name":"README.md","path":"examples/eks-cluster-with-iam-role-mappings/README.md","sha":"67c3a556cedc312b67bd50f8f439dff7bbad0034"},{"name":"dependencies.tf","path":"examples/eks-cluster-with-iam-role-mappings/dependencies.tf","sha":"9652dab961175e0f2273b109b5f1724a38e3970f"},{"name":"main.tf","path":"examples/eks-cluster-with-iam-role-mappings/main.tf","sha":"626919c788219b464259e939f376b6e5398c9e0c"},{"name":"outputs.tf","path":"examples/eks-cluster-with-iam-role-mappings/outputs.tf","sha":"e641213ad585fffe0f165b9a543d54c9175a1a7d"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/eks-cluster-with-iam-role-mappings/user-data/user-data.sh","sha":"b10c34bfe4c9d10101472b47edbc3b7dff42a88e"}]},{"name":"variables.tf","path":"examples/eks-cluster-with-iam-role-mappings/variables.tf","sha":"31d867ec870a904edd44951333c9fce45df49b55"}]},{"name":"eks-cluster-with-supporting-services","children":[{"name":"README.md","path":"examples/eks-cluster-with-supporting-services/README.md","sha":"e16f6e8fb05811e7425866525189b5cd209a0527"},{"name":"core-services","children":[{"name":"README.md","path":"examples/eks-cluster-with-supporting-services/core-services/README.md","sha":"c1eb41e7cc60a67d29ef846daf3b2e974ca59e6e"},{"name":"dependencies.tf","path":"examples/eks-cluster-with-supporting-services/core-services/dependencies.tf","sha":"0176248910eed450c12b54d10e3d74c8702c17ca"},{"name":"main.tf","path":"examples/eks-cluster-with-supporting-services/core-services/main.tf","sha":"bed1531141f5f1e03ce7f7535408fdb6295b2757"},{"name":"outputs.tf","path":"examples/eks-cluster-with-supporting-services/core-services/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"examples/eks-cluster-with-supporting-services/core-services/variables.tf","sha":"d05361c127cb11c156cfd35f7e880bc006442227"}]},{"name":"eks-cluster","children":[{"name":"README.md","path":"examples/eks-cluster-with-supporting-services/eks-cluster/README.md","sha":"8a60a01004a93bbbf2091b730f0207f6dd2cc07e"},{"name":"dependencies.tf","path":"examples/eks-cluster-with-supporting-services/eks-cluster/dependencies.tf","sha":"58c85fb4cb629a91afe41602e56072c19905e79b"},{"name":"main.tf","path":"examples/eks-cluster-with-supporting-services/eks-cluster/main.tf","sha":"29c1e42cdc7e2b48cded84f1fb45c64297e6a38f"},{"name":"outputs.tf","path":"examples/eks-cluster-with-supporting-services/eks-cluster/outputs.tf","sha":"be23a13dd6f4063be394b8ca7358b631d50fab8a"},{"name":"user-data","children":[{"name":"app_worker_user_data.sh","path":"examples/eks-cluster-with-supporting-services/eks-cluster/user-data/app_worker_user_data.sh","sha":"c5fdd13d5bb04f765f1c90e9f12d23c48e94a252"},{"name":"core_worker_user_data.sh","path":"examples/eks-cluster-with-supporting-services/eks-cluster/user-data/core_worker_user_data.sh","sha":"0fa26153108b3d030ceeaae777aeb0a7e115404e"}]},{"name":"variables.tf","path":"examples/eks-cluster-with-supporting-services/eks-cluster/variables.tf","sha":"88330995ab24d8351941b41088f3232a418f6ed5"}]},{"name":"nginx-service","children":[{"name":"README.md","path":"examples/eks-cluster-with-supporting-services/nginx-service/README.md","sha":"0f6649ddb0cbb5aa80a5bc1f3318ea1fd5d0dc35"},{"name":"dependencies.tf","path":"examples/eks-cluster-with-supporting-services/nginx-service/dependencies.tf","sha":"0176248910eed450c12b54d10e3d74c8702c17ca"},{"name":"main.tf","path":"examples/eks-cluster-with-supporting-services/nginx-service/main.tf","sha":"db605685e89d5d8ea0b04ae09d52b4acd815270c"},{"name":"templates","children":[{"name":"values.yaml","path":"examples/eks-cluster-with-supporting-services/nginx-service/templates/values.yaml","sha":"298435e01df9fa495b15d512073c62662d292cd3"}]},{"name":"variables.tf","path":"examples/eks-cluster-with-supporting-services/nginx-service/variables.tf","sha":"36ea6f8a36b19e34dbeeb25ae7e5fcf30c956b0f"}]},{"name":"packer","children":[{"name":"README.md","path":"examples/eks-cluster-with-supporting-services/packer/README.md","sha":"6a974a7fd5da7ac13309d9e0c4aaba7bd8cb46c7"},{"name":"build.json","path":"examples/eks-cluster-with-supporting-services/packer/build.json","sha":"047ee8df89c28fed157590ece33fd9a6edd524d4"}]}]},{"name":"eks-fargate-cluster-with-irsa","children":[{"name":"README.md","path":"examples/eks-fargate-cluster-with-irsa/README.md","sha":"27e8ad7773480a5431283c43609e3c7a47632f23"},{"name":"dependencies.tf","path":"examples/eks-fargate-cluster-with-irsa/dependencies.tf","sha":"88e84376868ae8dfc7b90483aa0fffe1c9d1a9ae"},{"name":"main.tf","path":"examples/eks-fargate-cluster-with-irsa/main.tf","sha":"605b50ca573ade388c9984ad070139568167b97f"},{"name":"outputs.tf","path":"examples/eks-fargate-cluster-with-irsa/outputs.tf","sha":"f059d7b74ffbfb06a0868d6d0a5d1831c8f45f10"},{"name":"variables.tf","path":"examples/eks-fargate-cluster-with-irsa/variables.tf","sha":"b10159edfba1ea09142e86cf19e2365393b96bed"}]},{"name":"eks-fargate-cluster-with-supporting-services","children":[{"name":"README.md","path":"examples/eks-fargate-cluster-with-supporting-services/README.md","sha":"56fdb5ede862db1664b3faa7b6aa86c21365c1fa"},{"name":"core-services","children":[{"name":"README.md","path":"examples/eks-fargate-cluster-with-supporting-services/core-services/README.md","sha":"cde0ae405e4d73e9e39c67045fb82de8187a673d"},{"name":"dependencies.tf","path":"examples/eks-fargate-cluster-with-supporting-services/core-services/dependencies.tf","sha":"0176248910eed450c12b54d10e3d74c8702c17ca"},{"name":"main.tf","path":"examples/eks-fargate-cluster-with-supporting-services/core-services/main.tf","sha":"7abe32a8f077f3758452253cc09083e5b0b51e96"},{"name":"variables.tf","path":"examples/eks-fargate-cluster-with-supporting-services/core-services/variables.tf","sha":"c63e2fdb8d5aa91830db61224ce75ee814d6fa56"}]},{"name":"eks-cluster","children":[{"name":"dependencies.tf","path":"examples/eks-fargate-cluster-with-supporting-services/eks-cluster/dependencies.tf","sha":"c7d533db5e590f72eddbe987d0b5353c11b570e1"},{"name":"main.tf","path":"examples/eks-fargate-cluster-with-supporting-services/eks-cluster/main.tf","sha":"91c44add94f28514ed2be32755aff944dcf8b64b"},{"name":"outputs.tf","path":"examples/eks-fargate-cluster-with-supporting-services/eks-cluster/outputs.tf","sha":"edddf9a6ab6f5927db366689db79e1b91db9d8c8"},{"name":"variables.tf","path":"examples/eks-fargate-cluster-with-supporting-services/eks-cluster/variables.tf","sha":"db997562498616550592cbbace9e3c0172761998"}]},{"name":"nginx-service","children":[{"name":"dependencies.tf","path":"examples/eks-fargate-cluster-with-supporting-services/nginx-service/dependencies.tf","sha":"0176248910eed450c12b54d10e3d74c8702c17ca"},{"name":"main.tf","path":"examples/eks-fargate-cluster-with-supporting-services/nginx-service/main.tf","sha":"1ae7751069711726f7c38fafe60d63d0c5f59494"},{"name":"templates","children":[{"name":"values.yaml","path":"examples/eks-fargate-cluster-with-supporting-services/nginx-service/templates/values.yaml","sha":"655914f91177135cb7c5f15b62166cfc82a62a91"}]},{"name":"variables.tf","path":"examples/eks-fargate-cluster-with-supporting-services/nginx-service/variables.tf","sha":"d3c166441cdc556b0839930fbc281b7e8a1bd57f"}]}]},{"name":"eks-fargate-cluster","children":[{"name":"README.md","path":"examples/eks-fargate-cluster/README.md","sha":"a1013138cb6f63a7109f287e76030b7cc71f447e"},{"name":"dependencies.tf","path":"examples/eks-fargate-cluster/dependencies.tf","sha":"88e84376868ae8dfc7b90483aa0fffe1c9d1a9ae"},{"name":"main.tf","path":"examples/eks-fargate-cluster/main.tf","sha":"4e321e439c0d55d997f2b7cf568d075f55c281e8"},{"name":"outputs.tf","path":"examples/eks-fargate-cluster/outputs.tf","sha":"9fb0eacd494d51072898a36f4d110a6c6ad77f6b"},{"name":"terraform.tfvars.back","path":"examples/eks-fargate-cluster/terraform.tfvars.back","sha":"6cb73f75cc7828c6b3efdc2a9b1787f75ed276d1"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/eks-fargate-cluster/user-data/user-data.sh","sha":"b10c34bfe4c9d10101472b47edbc3b7dff42a88e"}]},{"name":"variables.tf","path":"examples/eks-fargate-cluster/variables.tf","sha":"d6b3bf762a815889fc0eaec8ff547bb40f7271df"}]}]},{"name":"modules","children":[{"name":"eks-alb-ingress-controller-iam-policy","children":[{"name":"README.md","path":"modules/eks-alb-ingress-controller-iam-policy/README.md","sha":"c87be2ee00f8f59403f827303915b5a70c602002"},{"name":"iampolicy.json","path":"modules/eks-alb-ingress-controller-iam-policy/iampolicy.json","sha":"11b345cb86734392f4efca98ccbf76761bb5e313"},{"name":"main.tf","path":"modules/eks-alb-ingress-controller-iam-policy/main.tf","sha":"a79f5a2e6a0ba72562c5a87182db516d8824ed21"},{"name":"outputs.tf","path":"modules/eks-alb-ingress-controller-iam-policy/outputs.tf","sha":"b551b0bcc6eb1b43bfff1606696566658564cfb4"},{"name":"variables.tf","path":"modules/eks-alb-ingress-controller-iam-policy/variables.tf","sha":"250152e6bfeb02a16bed4151ffc7156636db1bd9"}]},{"name":"eks-alb-ingress-controller","children":[{"name":"README.md","path":"modules/eks-alb-ingress-controller/README.md","sha":"3bfcd0485ea2239eb786564e74c1de0715f23b57"},{"name":"main.tf","path":"modules/eks-alb-ingress-controller/main.tf","sha":"f748eaa5ccedcb6725677d629ed5ea55a6abbde8"},{"name":"variables.tf","path":"modules/eks-alb-ingress-controller/variables.tf","sha":"9be2ce78e03c5eb73817caf1717e5780bb488a79"}]},{"name":"eks-aws-auth-merger","children":[{"name":"Dockerfile","path":"modules/eks-aws-auth-merger/Dockerfile","sha":"5ac1f9c32ce10ae515a41a9e339fab08bce6faea"},{"name":"README.adoc","path":"modules/eks-aws-auth-merger/README.adoc","sha":"07d666370cec8ff194314ce0ebd587e5104ace0d"},{"name":"aws-auth-merger","children":[{"name":"aws_auth_merger.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/aws_auth_merger.go","sha":"061e755e548fce9bbf7ae98a5c4c2a0f30c22d35"},{"name":"aws_auth_merger_test.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/aws_auth_merger_test.go","sha":"218a7dbb20c3e5ba80e6540156a81241360c6930"},{"name":"cli.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/cli.go","sha":"8c335463605f65db25d773271da67a19d4caea7b"},{"name":"debouncer.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/debouncer.go","sha":"2376c66675e380b636421c17ea573fa12a141dd0"},{"name":"debouncer_test.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/debouncer_test.go","sha":"e6cd8e44503b4dea3e81b26ff20db39c35c8c72e"},{"name":"go.mod","path":"modules/eks-aws-auth-merger/aws-auth-merger/go.mod","sha":"b277857ad6724e8c97d0dd92a2c5933b442bbb51"},{"name":"go.sum","path":"modules/eks-aws-auth-merger/aws-auth-merger/go.sum","sha":"15f42b5e796ccc75490f2875985916eeb2eafdfc"},{"name":"main.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/main.go","sha":"9061cabd38668a323387cf8b7252ed2b45cbf2b9"},{"name":"mapping.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/mapping.go","sha":"1d0a36862e4b4a24ddd6112b016acce54e643775"},{"name":"mapping_test.go","path":"modules/eks-aws-auth-merger/aws-auth-merger/mapping_test.go","sha":"02c91e45a128d6c306ec42a548ac9023df350e78"}]},{"name":"core-concepts.md","path":"modules/eks-aws-auth-merger/core-concepts.md","sha":"2da0061c35747e9f280f8a440adfc4534da40fa4"},{"name":"main.tf","path":"modules/eks-aws-auth-merger/main.tf","sha":"987677a4e0822a0ef7f9292b5fd8af932c0902f6"},{"name":"outputs.tf","path":"modules/eks-aws-auth-merger/outputs.tf","sha":"d733fb246403f97ac011cbedf3f1d2761badef82"},{"name":"variables.tf","path":"modules/eks-aws-auth-merger/variables.tf","sha":"d40919e45570204bace7349dc8fdeb77e28a9634"}]},{"name":"eks-cloudwatch-container-logs","children":[{"name":"README.md","path":"modules/eks-cloudwatch-container-logs/README.md","sha":"c82a192074e8554383330893426764bfe75bbfe3"}]},{"name":"eks-cluster-control-plane","children":[{"name":"README.md","path":"modules/eks-cluster-control-plane/README.md","sha":"a3b4b0a40376bd7809d4bc25385b04414a8015f1"},{"name":"control_plane_scripts","children":[{"name":"bin","children":[{"name":"control_plane_scripts_py27_env.pex","path":"modules/eks-cluster-control-plane/control_plane_scripts/bin/control_plane_scripts_py27_env.pex","sha":"a02c9440827aac48475673ed80106b8cc1376bb4"},{"name":"control_plane_scripts_py3_env.pex","path":"modules/eks-cluster-control-plane/control_plane_scripts/bin/control_plane_scripts_py3_env.pex","sha":"3b4950866dbf6ad90a029585521aa90ed3e8887c"}]},{"name":"build.sh","path":"modules/eks-cluster-control-plane/control_plane_scripts/build.sh","sha":"33b5e9231babdb0c2c0997b04a964c27b98a4e13"},{"name":"cleanup_cluster_resources","children":[{"name":"__init__.py","path":"modules/eks-cluster-control-plane/control_plane_scripts/cleanup_cluster_resources/__init__.py","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"global_vars.py","path":"modules/eks-cluster-control-plane/control_plane_scripts/cleanup_cluster_resources/global_vars.py","sha":"47920d25645a8c168f196beb76eb37da60055dd3"},{"name":"main.py","path":"modules/eks-cluster-control-plane/control_plane_scripts/cleanup_cluster_resources/main.py","sha":"21dfb38d1bf8f4d15a03da5e09ae3ba575eb4501"},{"name":"vpc.py","path":"modules/eks-cluster-control-plane/control_plane_scripts/cleanup_cluster_resources/vpc.py","sha":"71706baa0364e0aa386048653e11e029b1a04be9"}]},{"name":"control_plane_scripts_utils","children":[{"name":"__init__.py","path":"modules/eks-cluster-control-plane/control_plane_scripts/control_plane_scripts_utils/__init__.py","sha":"37d050d1afd8ebb0c9d6916cff61fa674e6ac8a3"},{"name":"project_logging.py","path":"modules/eks-cluster-control-plane/control_plane_scripts/control_plane_scripts_utils/project_logging.py","sha":"c29bfb0dfe0a3d4e04aeaabff0b2e58387ccf12b"}]},{"name":"dev_requirements.txt","path":"modules/eks-cluster-control-plane/control_plane_scripts/dev_requirements.txt","sha":"430b91474dc8220624012e70d8c2e43582f17161"},{"name":"requirements.txt","path":"modules/eks-cluster-control-plane/control_plane_scripts/requirements.txt","sha":"0ae8cdb74f4c793658c5dfdd13ce1ec723f7b2a1"}]},{"name":"dependencies.tf","path":"modules/eks-cluster-control-plane/dependencies.tf","sha":"15dd4fe621801bbda76ff246780ab89e11308af0"},{"name":"main.tf","path":"modules/eks-cluster-control-plane/main.tf","sha":"ffb53ad1760bdca58a0847a80c317d48516ed570"},{"name":"outputs.tf","path":"modules/eks-cluster-control-plane/outputs.tf","sha":"26cae08068d8dabd6cbd1fc252ad5f8890e7b403"},{"name":"templates","children":[{"name":"kubectl_config.tpl","path":"modules/eks-cluster-control-plane/templates/kubectl_config.tpl","sha":"083a5e914505363541190db3ee412d8d9e15b4ec"}]},{"name":"variables.tf","path":"modules/eks-cluster-control-plane/variables.tf","sha":"a4d1105ac6379af33811060fa81dfc55dead675c"}]},{"name":"eks-cluster-managed-workers","children":[{"name":"README.md","path":"modules/eks-cluster-managed-workers/README.md","sha":"a44255e58e4c5949e3216339358124593ae2bbae"},{"name":"main.tf","path":"modules/eks-cluster-managed-workers/main.tf","sha":"56bad0a77dcc6eda3cf529007d2f354ba89bc82b"},{"name":"outputs.tf","path":"modules/eks-cluster-managed-workers/outputs.tf","sha":"391b5aff36a080568d94aae450d00b78488fb2e4"},{"name":"variables.tf","path":"modules/eks-cluster-managed-workers/variables.tf","sha":"fbb0d0efade0cb20f388b3c0f9cfeebf4cd87ff3"}]},{"name":"eks-cluster-workers-cross-access","children":[{"name":"README.md","path":"modules/eks-cluster-workers-cross-access/README.md","sha":"6c4e50bda62acc6c06d836488ef54f7119f27aee"},{"name":"main.tf","path":"modules/eks-cluster-workers-cross-access/main.tf","sha":"30885a053867992d0c3ee3804ba6833ae463c116"},{"name":"outputs.tf","path":"modules/eks-cluster-workers-cross-access/outputs.tf","sha":"c6c7f7a89007c55be5470ffd639c05c3fb052ad7"},{"name":"variables.tf","path":"modules/eks-cluster-workers-cross-access/variables.tf","sha":"d64aab893b6e909416189e985f072dd8809dfa2f"}]},{"name":"eks-cluster-workers","children":[{"name":"README.md","path":"modules/eks-cluster-workers/README.md","sha":"b846d1233c8a490fcb1bb0e7581c274f92d1c978"},{"name":"dependencies.tf","path":"modules/eks-cluster-workers/dependencies.tf","sha":"8c24d6dc80a1c281112928cba406d8853a2f615c"},{"name":"main.tf","path":"modules/eks-cluster-workers/main.tf","sha":"1703682dca05a710ea40bc0b068b8fd72c31fb0f"},{"name":"outputs.tf","path":"modules/eks-cluster-workers/outputs.tf","sha":"15a01dabd1c0a11011e2488c4df1f43468312454"},{"name":"variables.tf","path":"modules/eks-cluster-workers/variables.tf","sha":"242cb3e29fca1a19e3213c94accc2ca6fc2088c9"}]},{"name":"eks-container-logs","children":[{"name":"README.md","path":"modules/eks-container-logs/README.md","sha":"3af3ffd83b5c0205f122ecba654dec888c6efd72","toggled":true},{"name":"main.tf","path":"modules/eks-container-logs/main.tf","sha":"5dcf6e52a7cccba5b5248f7bdc4561e348db62c9"},{"name":"outputs.tf","path":"modules/eks-container-logs/outputs.tf","sha":"7061ed458fec528c8b8b587291f0eccb4324fb72"},{"name":"variables.tf","path":"modules/eks-container-logs/variables.tf","sha":"b8bb95011f2d835ce27b7c8fd856ab15a528cce7"}],"toggled":true},{"name":"eks-iam-role-assume-role-policy-for-service-account","children":[{"name":"README.md","path":"modules/eks-iam-role-assume-role-policy-for-service-account/README.md","sha":"efbbbd70fea3661c662750768facb7950239ffa3"},{"name":"main.tf","path":"modules/eks-iam-role-assume-role-policy-for-service-account/main.tf","sha":"be2fefe5e1a29a2582d1dcdc0b700b74f198cfc9"},{"name":"outputs.tf","path":"modules/eks-iam-role-assume-role-policy-for-service-account/outputs.tf","sha":"c2910cec89910bb06a157311ac8c4bf72835dfe5"},{"name":"variables.tf","path":"modules/eks-iam-role-assume-role-policy-for-service-account/variables.tf","sha":"dc660ddf84158851145289f6036a0fc19fbf7ce4"}]},{"name":"eks-k8s-cluster-autoscaler-iam-policy","children":[{"name":"README.md","path":"modules/eks-k8s-cluster-autoscaler-iam-policy/README.md","sha":"a22e2264a296fe1bf00f2c8b2f72ae728d0277c3"},{"name":"main.tf","path":"modules/eks-k8s-cluster-autoscaler-iam-policy/main.tf","sha":"c743f0e3523119155e2f2a6434e6f634d659aaee"},{"name":"outputs.tf","path":"modules/eks-k8s-cluster-autoscaler-iam-policy/outputs.tf","sha":"8b6c4e1747b3fa6a88c6233ec87aa2f450dfd334"},{"name":"variables.tf","path":"modules/eks-k8s-cluster-autoscaler-iam-policy/variables.tf","sha":"be3db9023160b3754187f2f21ce77772b43ced53"}]},{"name":"eks-k8s-cluster-autoscaler","children":[{"name":"README.md","path":"modules/eks-k8s-cluster-autoscaler/README.md","sha":"a74848607c42fcef696f121c2506ace0b83ced87"},{"name":"main.tf","path":"modules/eks-k8s-cluster-autoscaler/main.tf","sha":"8b4d132363a9f4cd0e5a273119e077db6084cbda"},{"name":"templates","children":[{"name":"node_affinity.yaml","path":"modules/eks-k8s-cluster-autoscaler/templates/node_affinity.yaml","sha":"c6eaf8e94fa7c893857cc009df954443239a8fe0"},{"name":"values.yaml","path":"modules/eks-k8s-cluster-autoscaler/templates/values.yaml","sha":"081ab7f76dc4e753975b427d7cee51589965715d"}]},{"name":"variables.tf","path":"modules/eks-k8s-cluster-autoscaler/variables.tf","sha":"6afcaee051c2b9c48dccff0c18d00c56a38f5818"}]},{"name":"eks-k8s-external-dns-iam-policy","children":[{"name":"README.md","path":"modules/eks-k8s-external-dns-iam-policy/README.md","sha":"a33d41f9824e6270ef4573d6b7e22b394224689c"},{"name":"main.tf","path":"modules/eks-k8s-external-dns-iam-policy/main.tf","sha":"b346bd0324c30907dd62ac89f93fe9cc7799fd4d"},{"name":"outputs.tf","path":"modules/eks-k8s-external-dns-iam-policy/outputs.tf","sha":"21604a63b741b94ea9ebffd20b18772131020fcf"},{"name":"variables.tf","path":"modules/eks-k8s-external-dns-iam-policy/variables.tf","sha":"250152e6bfeb02a16bed4151ffc7156636db1bd9"}]},{"name":"eks-k8s-external-dns","children":[{"name":"README.md","path":"modules/eks-k8s-external-dns/README.md","sha":"0feaa9793a59843f8999449ffcb2a53e9d33a120"},{"name":"main.tf","path":"modules/eks-k8s-external-dns/main.tf","sha":"a4d504072bd2c45e64a9b23eac5e3bdc59c68887"},{"name":"templates","children":[{"name":"node_affinity.yaml","path":"modules/eks-k8s-external-dns/templates/node_affinity.yaml","sha":"c6eaf8e94fa7c893857cc009df954443239a8fe0"},{"name":"values.yaml","path":"modules/eks-k8s-external-dns/templates/values.yaml","sha":"c2b1092f2fcef1ac1dff0d4ace48e90364693139"}]},{"name":"variables.tf","path":"modules/eks-k8s-external-dns/variables.tf","sha":"0314d4675c3a06f8f4dbdd4c31eb6e170a30b8cf"}]},{"name":"eks-k8s-role-mapping","children":[{"name":"README.md","path":"modules/eks-k8s-role-mapping/README.md","sha":"2962e93307761b2356c62f0ac8068dc01f98d9f4"},{"name":"main.tf","path":"modules/eks-k8s-role-mapping/main.tf","sha":"28777db2a9c763ff30a55a362425a318ff3538ac"},{"name":"outputs.tf","path":"modules/eks-k8s-role-mapping/outputs.tf","sha":"95d4d4ec652bb541b91a2844e00f68064b423e60"},{"name":"variables.tf","path":"modules/eks-k8s-role-mapping/variables.tf","sha":"5c696a015d9e8b44e5687b758c61ac49f3a4b7ef"}]},{"name":"eks-scripts","children":[{"name":"README.md","path":"modules/eks-scripts/README.md","sha":"96baaf535647b9f4c364d6a19057bcccb42df2be"},{"name":"bin","children":[{"name":"map-ec2-tags-to-node-labels","path":"modules/eks-scripts/bin/map-ec2-tags-to-node-labels","sha":"8087c82d4d47f25439f118c2a51e59d22689ada7"},{"name":"map_ec2_tags_to_node_labels.py","path":"modules/eks-scripts/bin/map_ec2_tags_to_node_labels.py","sha":"f75ad19587e95b2bd8924125ea2a1a697154909f"}]},{"name":"dev_requirements.txt","path":"modules/eks-scripts/dev_requirements.txt","sha":"f56f9d1629a85734fe16ed70f00f36b830cd97c9"},{"name":"install.sh","path":"modules/eks-scripts/install.sh","sha":"7f192fca97b098482a8a398019d4d53f45dba478"}]},{"name":"eks-vpc-tags","children":[{"name":"README.md","path":"modules/eks-vpc-tags/README.md","sha":"b53e923baaa79718b55a272158ff9b710871a6ce"},{"name":"outputs.tf","path":"modules/eks-vpc-tags/outputs.tf","sha":"0ef2787cfd02ea8668c687302b1929618079a0b2"},{"name":"variables.tf","path":"modules/eks-vpc-tags/variables.tf","sha":"a6e332e9da4e473e1e42b1ca6c7b0ba139a77cfb"},{"name":"versions.tf","path":"modules/eks-vpc-tags/versions.tf","sha":"e5d003c3e7a7296ca0f610fc77f94f2139fc59d2"}]}],"toggled":true},{"name":"rfc","children":[{"name":"shipping-logs-to-cloudwatch.md","path":"rfc/shipping-logs-to-cloudwatch.md","sha":"3ac6a0fd509477c36e1b4079e82ed3def7fe03d8"}]},{"name":"setup.cfg","path":"setup.cfg","sha":"981bc2bfd0b35029438d56c6d862a7f1519b8fe6"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"9bf8180d731bdc892279fcdbcbb03d245f31f83a"},{"name":"eks_cluster_integration_test.go","path":"test/eks_cluster_integration_test.go","sha":"471a3100ec1db2b42e2f80f2184621b3553020f3"},{"name":"eks_cluster_managed_workers_test.go","path":"test/eks_cluster_managed_workers_test.go","sha":"9d9e7fe7f43d0a3e8796780bb5414f50fbc23356"},{"name":"eks_cluster_test_helpers.go","path":"test/eks_cluster_test_helpers.go","sha":"8fbf2ff0ea963d8a97485b2bda0240f47f98e751"},{"name":"eks_cluster_upgrade_test.go","path":"test/eks_cluster_upgrade_test.go","sha":"9aa60afd729ab24d4cf9d6c65b7fc0339ad5ab0b"},{"name":"eks_cluster_with_auth_merger_test.go","path":"test/eks_cluster_with_auth_merger_test.go","sha":"0873718a4f5494698d271239b529d29d6df4f255"},{"name":"eks_cluster_with_iam_role_test.go","path":"test/eks_cluster_with_iam_role_test.go","sha":"6bdef141d0caef255342a74de00ab8e1088c0610"},{"name":"eks_cluster_with_supporting_services_test.go","path":"test/eks_cluster_with_supporting_services_test.go","sha":"d1e60137ff053fa816c11fd4cb10e73c61b41126"},{"name":"eks_cluster_workers_optional_test.go","path":"test/eks_cluster_workers_optional_test.go","sha":"1b2dadebcef8d451b311c20e24c89c256080c6e8"},{"name":"eks_envelope_encryption_test.go","path":"test/eks_envelope_encryption_test.go","sha":"3d8b92c4d3d4244c6431ccae95f0faeb0328bdce"},{"name":"eks_fargate_cluster_disable_public_endpoint_test.go","path":"test/eks_fargate_cluster_disable_public_endpoint_test.go","sha":"25ba0984ef5979ca146d16b63654559939d822db"},{"name":"eks_fargate_cluster_irsa_test.go","path":"test/eks_fargate_cluster_irsa_test.go","sha":"8a1e4f4599f63b357af1ca36b135898e7663f1fe"},{"name":"eks_fargate_cluster_public_access_cidr_test.go","path":"test/eks_fargate_cluster_public_access_cidr_test.go","sha":"2a82ad5a0bbb9311bb9c91a2c0be3f3dbe1b4d5e"},{"name":"eks_fargate_cluster_test.go","path":"test/eks_fargate_cluster_test.go","sha":"a50d3691cbdec0ba41e2212015105254d7a516c7"},{"name":"eks_fargate_cluster_with_supporting_services_test.go","path":"test/eks_fargate_cluster_with_supporting_services_test.go","sha":"a236dc2c1647da144a3fa973492b18ad80d64103"},{"name":"eks_mixed_cluster_dns_test.go","path":"test/eks_mixed_cluster_dns_test.go","sha":"069332615ab046026f91262ebfb3715786132895"},{"name":"errors.go","path":"test/errors.go","sha":"be062fe0205ff82db8183d0fde639aa1883013ad"},{"name":"go.mod","path":"test/go.mod","sha":"d5a1eed94e3e1496d6d299c36c5c0a2016a84f95"},{"name":"go.sum","path":"test/go.sum","sha":"7844bf26994c49320e11604a6ebb2b32afeecc6b"},{"name":"kubefixtures","children":[{"name":"autoscaler-test-pods-deployment.yml","path":"test/kubefixtures/autoscaler-test-pods-deployment.yml","sha":"b2d94c4bfa729b639290ee21629c19ca6ea694ee"},{"name":"eks-irsa-test.yml","path":"test/kubefixtures/eks-irsa-test.yml","sha":"db5439cf6d38873dbae71daa4197d6947990a94a"},{"name":"eks-k8s-role-mapping-test-role.yml","path":"test/kubefixtures/eks-k8s-role-mapping-test-role.yml","sha":"ede7587308d2a4ecf55042b05800099c43f3af7d"},{"name":"kube-system-sa-admin-binding.yml","path":"test/kubefixtures/kube-system-sa-admin-binding.yml","sha":"282d406512102cbe54e952575f26e7e0fbb2aa9a"},{"name":"nginx-deployment.yml","path":"test/kubefixtures/nginx-deployment.yml","sha":"a58866e59c113635af24982cfb0b530f0c416af0"},{"name":"robust-nginx-deployment.yml","path":"test/kubefixtures/robust-nginx-deployment.yml","sha":"87ead0f9733e422099bc430ed281e2054e698f10"}]},{"name":"script_tests","children":[{"name":"executor.sh","path":"test/script_tests/executor.sh","sha":"458c534996fbc045081d1cfae521c090f6787a7f"},{"name":"requirements.txt","path":"test/script_tests/requirements.txt","sha":"26025d496c31cfde48e787907ce830ee446f5b05"},{"name":"test_map_ec2_tags_to_node_labels.py","path":"test/script_tests/test_map_ec2_tags_to_node_labels.py","sha":"1bb3a5eae3727c0e6caf29c2cf4b7d596bb9a161"},{"name":"tox.ini","path":"test/script_tests/tox.ini","sha":"a7b8c79ca45e700e9cb7b8b493b37c68bc4408c2"}]},{"name":"terratest_options.go","path":"test/terratest_options.go","sha":"008ea2e33414f97f64672602a6451a22d43df894"},{"name":"test_debug_helpers.go","path":"test/test_debug_helpers.go","sha":"c71a7a9d5b68f0f59d2518496d9f5893206b5e22"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"4633eab64d3382614ac38d1f3840e45c522486e2"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"eks-container-logs-module\">EKS Container Logs Module</h1><div class=\"preview__body--border\"></div><p>This Terraform Module installs and configures\n<a href=\"https://github.com/aws/aws-for-fluent-bit\" class=\"preview__body--description--blue\" target=\"_blank\">aws-for-fluent-bit</a> on an EKS cluster, so that\neach node runs <a href=\"https://fluentbit.io/\" class=\"preview__body--description--blue\" target=\"_blank\">fluent-bit</a> to collect the logs and ship to CloudWatch Logs, Kinesis Streams, or\nKinesis Firehose.</p>\n<p>This module uses the community helm chart, with a set of best practices inputs.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-does-this-work\">How does this work?</h2>\n<p>This module solves the problem of unifying the log streams in your Kubernetes cluster to be shipped to an aggregation\nservice on AWS (CloudWatch Logs, Kinesis, or Firehose) so that you have a single interface to search and monitor your\nlogs. To achieve this, the module installs a service (<code>fluent-bit</code>) that monitors the log files on the filesystem,\nparses custom log formats into a unified format, and ships the result to a centralized log aggregation service\n(CloudWatch).</p>\n<p><code>fluent-bit</code> is installed as a Kubernetes\n<a href=\"https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/\" class=\"preview__body--description--blue\" target=\"_blank\"><code>DaemonSet</code></a>, which ensures that there is one\n<code>fluent-bit</code> <code>Pod</code> running per node. In this way, we are able to ensure that all workers in the cluster are running the\n<code>fluent-bit</code> service for shipping the logs into CloudWatch.</p>\n<p>You can read more about <code>fluent-bit</code> in their <a href=\"https://fluentbit.io/\" class=\"preview__body--description--blue\" target=\"_blank\">official home page</a>. You can also learn more about\nCloudWatch logging in the <a href=\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html\" class=\"preview__body--description--blue\" target=\"_blank\">official AWS\ndocs</a>.</p>\n<h2 class=\"preview__body--subtitle\" id=\"what-is-the-difference-with-fluentd\">What is the difference with fluentd?</h2>\n<p><a href=\"https://fluentbit.io/\" class=\"preview__body--description--blue\" target=\"_blank\">fluent-bit</a> is an optimized version of <a href=\"https://www.fluentd.org/\" class=\"preview__body--description--blue\" target=\"_blank\">fluentd</a> that focuses on\nstreaming and aggregating log files. <code>fluentd</code> has a larger ecosystem of plugins that enable various processing\ncapabilities on top of the logs prior to aggregating in the data store.</p>\n<p>For most EKS deployments, it is recommended to use this <code>fluent-bit</code> module for container log aggregation. Unless you have a specific\nneed for a plugin only supported by <code>fluentd</code>, the superior performance and memory footprint of <code>fluent-bit</code> will\nensure resources are available on your EKS workers for your Pods.</p>\n<h2 class=\"preview__body--subtitle\" id=\"log-format\">Log format</h2>\n<p>This module leverages native plugins for Kubernetes built into <code>fluent-bit</code> that extract additional\nmetadata for each Pod that is reporting. Each log is shipped to the respective outputs in the following structure:</p>\n<pre>{\n <span class=\"hljs-attr\">\"kubernetes\"</span>: {\n <span class=\"hljs-attr\">\"namespace_name\"</span>: <span class=\"hljs-string\">\"NAMESPACE_WHERE_POD_LOCATED\"</span>,\n <span class=\"hljs-attr\">\"pod_name\"</span>: <span class=\"hljs-string\">\"NAME_OF_POD_EMITTING_LOG\"</span>,\n <span class=\"hljs-attr\">\"pod_id\"</span>: <span class=\"hljs-string\">\"ID_IN_KUBERNETES_OF_POD\"</span>,\n <span class=\"hljs-attr\">\"container_hash\"</span>: <span class=\"hljs-string\">\"KUBERNETES_GENERATED_HASH_OF_CONTAINER_EMITTING_LOG\"</span>,\n <span class=\"hljs-attr\">\"container_name\"</span>: <span class=\"hljs-string\">\"NAME_OF_CONTAINER_IN_POD_EMITTING_LOG\"</span>,\n <span class=\"hljs-attr\">\"docker_id\"</span>: <span class=\"hljs-string\">\"ID_IN_DOCKER_OF_CONTAINER\"</span>,\n <span class=\"hljs-attr\">\"host\"</span>: <span class=\"hljs-string\">\"NODE_NAME_OF_HOST_EMITTING_LOG\"</span>,\n <span class=\"hljs-attr\">\"labels\"</span>: {\n <span class=\"hljs-attr\">\"KEY\"</span>: <span class=\"hljs-string\">\"VALUE\"</span>,\n },\n <span class=\"hljs-attr\">\"annotations\"</span>: {\n <span class=\"hljs-attr\">\"KEY\"</span>: <span class=\"hljs-string\">\"VALUE\"</span>\n }\n },\n <span class=\"hljs-attr\">\"log\"</span>: <span class=\"hljs-string\">\"CONTENTS_OF_LOG_MESSAGE\"</span>,\n <span class=\"hljs-attr\">\"stream\"</span>: <span class=\"hljs-string\">\"STDERR_OR_STDOUT\"</span>,\n <span class=\"hljs-attr\">\"time\"</span>: <span class=\"hljs-string\">\"TIMESTAMP_OF_LOG\"</span>\n}\n</pre>\n<p>This allows you to filter and search the logs by the respective attributes. For example, the following CloudWatch\nInsights Query can be used to search for all logs from Pods in the <code>kube-system</code> Namespace:</p>\n<pre>fields @timestamp, @message\n|<span class=\"hljs-built_in\"> filter </span>kubernetes.namespace_name = <span class=\"hljs-string\">\"kube-system\"</span>\n| sort @timestamp desc\n| limit 20\n</pre>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<ul>\n<li>See the <a href=\"/repos/v0.31.3/terraform-aws-eks/README.adoc\" class=\"preview__body--description--blue\">root README</a> for instructions on using Terraform modules.</li>\n<li>See the <a href=\"/repos/v0.31.3/terraform-aws-eks/examples/eks-cluster-with-supporting-services\" class=\"preview__body--description--blue\">eks-cluster-with-supporting-services example</a> for example\nusage.</li>\n<li>See <a href=\"/repos/v0.31.3/terraform-aws-eks/modules/eks-container-logs/variables.tf\" class=\"preview__body--description--blue\">variables.tf</a> for all the variables you can set on this module.</li>\n<li>See <a href=\"/repos/v0.31.3/terraform-aws-eks/modules/eks-container-logs/outputs.tf\" class=\"preview__body--description--blue\">outputs.tf</a> for all the variables that are outputed by this module.</li>\n<li>This module uses <a href=\"https://www.terraform.io/docs/providers/helm/index.html\" class=\"preview__body--description--blue\" target=\"_blank\">the <code>helm</code> provider</a>.</li>\n</ul>\n<h2 class=\"preview__body--subtitle\" id=\"prerequisites\">Prerequisites</h2>\n<h3 class=\"preview__body--subtitle\" id=\"helm-setup\">Helm setup</h3>\n<p>This module uses <a href=\"https://helm.sh/docs/\" class=\"preview__body--description--blue\" target=\"_blank\"><code>helm</code> v3</a> to deploy <code>fluentd</code> to the Kubernetes cluster.</p>\n","repoName":"terraform-aws-eks","repoRef":"v0.29.0","serviceDescriptor":{"serviceName":"EC2 Kubernetes Service (EKS) Cluster","serviceRepoName":"terraform-aws-eks","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a Kubernetes cluster on top of Amazon EC2 Kubernetes Service (EKS).","imageUrl":"eks.png","licenseType":"subscriber","technologies":["Terraform","Python","Bash"],"compliance":[],"tags":[""]},"serviceCategoryName":"Docker orchestration","fileName":"README.md","filePath":"/modules/eks-container-logs","title":"Repo Browser: EC2 Kubernetes Service (EKS) Cluster","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
main.tf
