Chef Workflow Delivery Review Error Uploading to Supermarket Invalid Json

My visitor's Chef Automate workflow pipelines were designed as a part of a middleware infrastructure project. The project required three motorcar scaled instances each sitting backside their ain AWS ELB. The projection enlisted the services of five teams, each with their ain specialization for the project. The Infrastructure squad created AWS Cloudformation Templates (CFTs), CFT cookbooks, VPCs, security groups and ELBs. The middleware squad created the cookbooks for the respective instances including the underlying base of operations cookbooks which will be utilized by or company for future projects. The QA team created and iterated upon smoke and functional testing for single instances and their communication between other instances. Finally, the Security team determined the compliance testing necessary for instances' and helped create proper security testing which would end pipelines should servers fall out of compliance.

When designing the infrastructure and procedures for my visitor'south Chef Automate workflow pipelines we came across a number of hurdles.

Starting time, when provisioning instances via our CFT cookbook, the nodes are bootstrapped with chef client with a user data script. After chef client is installed, via the script, the nodes will run their first-boot.json. This contains the name of the cookbook for the the current project pipeline. If the recipe fails, however, during the initial bootstrapping process the node volition not be attached appropriately to chef server.

This bootstrapping process is a necessary component for AutoScaled instances. If new instances are booted, as a part of an AutoScale process, those nodes will require a bootstrap process be run with the latest cookbooks. Therefore, testing of the cookbook will need to be independent of the CFT deployment steps.

In guild to bypass this result my company developed a pipeline that calls on, non only, our internal CFT provisioning cookbook but also test kitchen for our acceptance nodes.

By using kitchen-ec2 we are able to converge and destroy our cookbooks in acceptance to verify their viability before passing them to our user data script. This is fabricated easier with the inclusion of the delivery-sugar cookbook. Delivery-sugar contains resource that allow for the cosmos, convergence and destruction of EC2, Azure, Docker and Vsphere instances using the delivery_test_kitchen resource.

My company is currently calling on kitchen-ec2 for instance cosmos. EC2 currently requires ALL of the following components to run successfully.

Examination Kitchen Setup (Acceptance Stage Provisioning):

In society to enable this functionality please perform the following prerequisite steps.

Add ALL of the post-obit items to the advisable data pocketbook within your Chef Server

Yous tin convert the private cardinal content to a JSON-compatible string with the post-obit control.

            chef exec ruby -e 'p ARGF.read' automate_kitchen.pem >> automate_kitchen.json                      

Since the individual fundamental should be secured this data bag should be encrypted. In guild to add an encrypted databag to the chef server yous must first accept proper admission to the chef server, which is necessary for a knife command to exist run. Afterward this permission is in identify yous must run the following command.

          knife data pocketbook create delivery-secrets                          -                  -                                              --secret-file encrypted_data_bag_secret                                                                                                  

Where

is the name of your enterprise, is the name of your organization and is the current proper name of the pipeline you are creating.

In society to decrypt this information the encrypted_data_bag_secret file, used to encrypt the information bag, must be added to your Chef Build servers at the following location.

          /etc/chef/        

Once these components are deployed Customize your kitchen YAML file with all the required information needed past the kitchen-ec2 commuter driver.

NOTE  This kitchen.yml file will be the one found in your .delivery/build_cookbook and not the 1 establish under your project cookbook

Delivery-sugar volition expose the following ENV variables for utilize by kitchen.

• KITCHEN_INSTANCE_NAME - ready to the - values provided by delivery-cli
• KITCHEN_EC2_SSH_KEY_PATH - path to the SSH private key created from the delivery-secrets data purse created from the step in a higher place.

These variable may be used in your kitchen YAML similar the following case:

One time the prerequisites are in identify you tin can use delivery_test_kitchen within your .delivery/build_cookbook/provision.rb to deploy instances through exam kitchen.

Trigger a kitchen converge and destroy action using Ec2 driver and pointing information technology to .kitchen.ec2.yml in delivery.

NOTE: When adding a repo_path my companychooses #{workflow_workspace_repo}/.delivery/build_cookbook/. This is by preference and the location of the .yml file can sit wherever the user requires.

Trigger a kitchen create passing extra options for debugging

Trigger a kitchen create extending the timeout to twenty minutes.


Since nosotros are only using kitchen in our acceptance node my visitor must add the following logic to verify test kitchen is not used outside of the acceptance stage. (workflow_stage is a resource provided past delivery-carbohydrate)

Version Pinning

The 2nd consequence we were presented with in creating our workflow pipelines came in the pinning of our environments.

If using base cookbooks for multiple projects, pinning should not exist done on the base of operations cookbook itself. Since cookbooks are pinned at an environment level if the base of operations cookbook is pinned at the surround and and so updated, that base of operations cookbook update will in issue modify all projects using it in that environs (credence, union, rehearsal delivered. To forestall this pinning from taking identify, through workflow, under

            .commitment/build-cookbook/provision.rb                      

comment out

            commitment-truck::provision          

In turn if nosotros version pin only the part cookbook at the environment level, being project specific, any changes made to the function cookbook should not have an effect on any other project.

This does hateful that in order for a base cookbook to exist updated in a project its version must be changed in the role cookbook. And so for every underlying cookbook modify the role cookbook will demand to be version bumped. This is a much more manual procedure, only it will provide protection from projects breaking with a modify to ane base of operations cookbook.

This also has the added benefit of version controlling whatever version bumps we have in our environments for a given projects node. Since the only version pins in an environs fall on the role cookbook, all other changes to versions should be controlled through the function cookbooks metadata and delivery cli commands. These commits tin be tied back to individual users and version changes which will better stabilize the environments.

The leading measure in Workflow, if base cookbooks are not projection specific, should sit with office cookbooks. These cookbooks should exist used to provision servers, and version pin underlying cookbooks, when going through the Union, Rehearsal and Delivered stages of the Chef Automate Workflow to separate projection version pinning.

Setting up Metadata.rb, Provision.rb, Kitchen.yml and Berksfile in .delivery/build_cookbook

Note: before adding the workflow provisioning steps to the build_cookbook please add the project cookbook to the chef server, Either through automate workflow or through a pocketknife control. If the project cookbook is not available upon the get-go run of the pipeline it volition fail when trying to download cookbooks

With these two issues resolved, and explained, it is now time to setup the rest of our workflow pipeline.

We volition beginning by modifying our Berksfile within .delivery/build_cookbook/. Since we volition be calling on cookbooks that are currently stored in the chef server we will need to brand sure that the workflow pipeline can reach out to it to observe cookbooks. We exercise this by adding the chef server source

Side by side, nosotros will change our metadata.rb file. We need to make sure we are calling in delivery-sugar, commitment-truck, the current project cookbook for the pipeline and the cookbook we are using to provision our servers.

Notation: We only need to call the provisioning cookbook here if this is the function cookbook

We volition also configure our kitchen.yml (which nosotros have named here every bit kitchen.ec2.yml) as we described in the steps above. This file is used for our kitchen converge and destroy in our credence provisioning stage.

Annotation: practice not forget to change the cookbook we are calling in the kitchen.yml to reflect the electric current cookbook nosotros are sitting in. (Come across run_list)

Finally, we will modify our provision.rb file. Depending on our surroundings (role cookbook vs base cookbook or wrapper cookbook. Please see the documentation for version pinning for further explanation).

In a ROLE cookbook We volition telephone call upon the provisioning cookbook if nosotros are in the union, rehearsal or delivered stage. This bank check can be made using the delivery-sugar resource workflow_stage which volition call the electric current stage the pipeline is currently running in.

We will also call on the commitment-truck::provision cookbook to pin our surroundings.

NOTE: the delivery-truck::provision recipe is included AFTER the run of our provisioning cookbook, See the section on versioning for more than information)

If we are NOT IN A ROLE COOKBOOK delivery-truck::provision volition not be chosen. We will besides not need to include the recipe for provisioning in marriage, rehearsal or delivered. To keep things simple, and to prevent us from having to make too many modifications to our code, we will simply add a warning message in place of the provisioning cookbook includes.

Once these changes are saved nosotros tin can version crash-land our projection cookbook, either through the metadata.rb file or the delivery command, and run delivery review.

Note: this version bump is washed in the PROJECT COOKBOOK not the build cookbook.

This will push the cookbook into Automate and kickoff the Chef Automate Workflow Pipeline.

barnettawase1936.blogspot.com

Source: https://www.servernameactive.com/search/label/recipes?max-results=6

Share this post