CircleCI Blog Running regular security scans with scheduled pipelines

Security is a vital part of application development, yet it may be neglected until an attacker takes advantage of a vulnerability in the system. The consequences of a security breach can damage an application's integrity as well as a company's reputation and revenue. Software architects and engineers need to pay special attention to securing the systems they work on.Running security-based tests and automating security into the CI/CD workflow is the inspiration behind DevSecOps, an extension of the DevOps methodology into the field of application security.In this article, I will lead you through automating security checks in an application. I will show you how to run a test on a web form that injects the URL field with malicious domains. The goal of the test is to break the web form. To stop the attack, the web form has been protected with logic that checks to see if a user is trying to enter a malicious domain. I will show you how to use scheduled pipelines to run the security tests on a regular schedule.PrerequisitesTo follow along with this tutorial, you will need these in place:Node.js installed on your system (version 12 or greater)Git installed on your systemA Webshrinker account; sign up for a free trial hereA CircleCI accountOnce you have these set up, you can begin.Getting Webshrinker credentialsWebshrinker is an AI-powered domain categorization system owned by DNS security company DNSFilter. Webshrinker is capable of identifying threat domains and labeling them according to their threat categories. Phishing, malware, and botnet are just three of the threat types Webshrinker can identify.The form we are protecting in the project takes a fully qualified domain name in its URL field. It sends the domain name to the Webshrinker API to be scanned for threats. If the results indicate that it is malicious, Webshrinker returns a threat identifier. The form uses the threat identifier to deny the processing of the domain entry.You will need an API key and API secret to use Webshrinker in this demo. Once you have an account created, you can create new API credentials by going to the API Access Keys page and clicking the Create API Key button. That generates your API secret and token.You will use your credentials in the next step.Cloning and running the demo applicationThe first step is to run the demo web form locally and inspect its behavior. You can clone the code for the form from this repository by running this command from anywhere on your system:git clone -b base-project --single-branch https://github.com/coderonfleek/scheduled-security-scan.gitOnce you have the code on your system, install the dependencies by running this command:cd scheduled-security-scan npm installNow go to lines 65 and 66 in the index.html file at the root of the project and replace the placeholders with your API key and API secret, respectively.When the installation of dependencies is complete, run the application with this command:npm startThe application boots up, and you can view the web form at https://localhost:5000.Enter an email address into the email field. In the URL field, enter a safe domain like facebook.com and click Submit. You will get a safe response on the right side of the form.Now, test the URL field of the form using this threat-based domain name: selcdn.ru. Please, do not enter this domain name directly in your browser. You will get a threat alert message.Adding the security testsThe next step is to write tests that perform the security checks automatically. You will be adding some automated functional tests using Google's Puppeteer. Puppeteer simulates the way a real-world user would fill the form.Add a new file named login.test.js at the root of the project and enter this code:const puppeteer = require("puppeteer"); const user_email = "test@example.com"; const non_threat_site = "facebook.com"; const malicious_site = "selcdn.ru"; const phishing_site = "mail.glesys.se"; const expected_safe_site_message = "Entry clean, process form"; const expected_threat_site_message = "Threat Detected. Do not Process"; test("Check Non-threat Site Entry", async () => { const browser = await puppeteer.launch(); try { const page = await browser.newPage(); await page.goto("http://localhost:5000"); await page.type("#userEmail", user_email); await page.type("#userSite", non_threat_site); await page.click("#submitButton"); let messageContainer = await page.$("#infoDisplay"); await page.waitForTimeout(4000); let value = await messageContainer.evaluate((el) => el.textContent); console.log(value); expect(value).toBe(expected_safe_site_message); } finally { await browser.close(); } }, 120000); test("Check Malicious Site Entry", async () => { const browser = await puppeteer.launch(); try { const page = await browser.newPage(); await page.goto("http://localhost:5000"); await page.type("#userEmail", user_email); await page.type("#userSite", malicious_site); await page.click("#submitButton"); let messageContainer = await page.$("#infoDisplay"); await page.waitForTimeout(4000); let value = await messageContainer.evaluate((el) => el.textContent); console.log(value); expect(value).toBe(expected_threat_site_message); } finally { await browser.close(); } }, 120000);This file contains two test cases. The first checks that non-threat domains like facebook.com are not blocked by the system. This keeps your security implementation from overzealously blocking regular domains.The second test case checks for malicious entries by using a sample malicious domain. If the form blocks this domain, the test passes. If the malicious domain is not blocked, the test fails.Save the file and go to the root of the project in your terminal. Make sure that the application is running in another shell; the tests need it to run successfully. Run this command:npm run testOnce the test run is complete, you will have results in your CLI like these:MACs-MBP-2:scheduled-security-scan mac$ npm run test > xss-attack@1.0.0 test /Users/mac/Documents/CircleCiProjects/2022/scheduled-security-scan > jest console.log Entry clean, process form at Object.<anonymous> (login.test.js:26:13) PASS ./login.test.js (109.034 s) ✓ Check Non-threat Site Entry (80318 ms) ✓ Check Malicious Site Entry (8210 ms) Test Suites: 1 passed, 1 total Tests: 2 passed, 2 total Snapshots: 0 total Time: 120.252 s Ran all test suites. console.log Threat Detected. Do not Process at Object.<anonymous> (login.test.js:54:13)Developers can be tempted to write tests that match their code capabilities. As a best practice, be sure to separate the development and testing teams. This practice lets the testing team write exhaustive tests and protects developers from themselves.Adding the pipeline configuration scriptTo automate the testing process, we can build a continuous integration (CI) pipeline using CircleCI.To set up the CI pipeline, you need a pipeline configuration script. You can add this script into a .circleci/config.yml file. Create the file at the root of the project and enter:version: 2.1 jobs: build: working_directory: ~/repo docker: - image: cimg/node:14.18-browsers steps: - checkout - run: name: Update NPM command: "sudo npm install -g npm" - restore_cache: key: dependency-cache-{{ checksum "package-lock.json" }} - run: name: Install Dependencies command: npm install - save_cache: key: dependency-cache-{{ checksum "package-lock.json" }} paths: - ./node_modules - run: name: Run the application command: node server.js background: true - run: name: Run tests command: npm run testThis configuration script:Pulls in a Docker image with browsers and Node.js installedUpdates npm and installs the required project dependenciesRuns the application in the background so that the Cypress tests can load itRuns the testsSave the file and push the project to GitHubNext, add the repository as a CircleCI project.Once the project is set up on CircleCI, the tests run immediately. When the build is done, you will get a successful build status.Click the Build link to review the build details.Setting up a scheduled pipeline on CircleCIUsually, your CI pipeline runs only when a new commit is made to the remote code repository you used when you set up the pipeline. To get the most from security scans like the tests we have written, the security state of the application should be updated even when no new code has been pushed.That means our pipeline needs to run at regular intervals, like a cron job, to accurately report the security status of the application. To keep things efficient, it is better not to mix security tests with functionality tests that check for bugs or validate application functions.To get your security tests running automatically and on schedule, CircleCI's scheduled pipelines technology is the way to go.With scheduled pipelines on CircleCI, you can configure your pipelines to run on just one specific day or on all days of the week at specific times.For this tutorial, we will set up our pipeline to run every five minutes every day of the week. The interval set for this project is not based on any real-life situation, detailed consideration, or best practice. It is just to demonstrate the pipeline running periodically during the demo period.To configure your pipeline to run on a schedule, go to CircleCI and select Project Settings, then Triggers.On the Triggers page, click Add Scheduled Trigger to display the trigger form. Fill in the form to configure the pipeline to run every five minutes, every day.Click Save Trigger and your trigger will be created.Return to the Pipelines page and wait for at least five minutes. Your pipeline will be triggered by the Cir