Write Code for Batch Jobs

Overview

Previously, developers required LeadSquared administrator access to create and test Batch Jobs. The values of confidential variables (e.g., access keys, secret keys, FTP credentials, SQL server credentials, etc.) and critical operations like unpublishing and deleting Lapps were made available through administrator access.

The Batch Jobs SDK now helps mitigate security risks by enabling developers to create, test, and publish Batch Jobs (on their local machines), without administrator access to your LeadSquared account. It grants developers limited access to Batch Jobs functionality through distinct access levels (Administrator, Developer – Configurator, Developer) and token-based authorization.

The Batch Jobs SDK package facilitates quick and easy offline development. This guide documents the features available in the SDK, and how they work. For more information, see lsbatch Software Development Kit.

 

Developer Access Levels

The following access levels and permissions are available –

Create Batch Job Upload Code Add Variable Publish  *Schedule
Admin Yes Yes Yes Yes No
Developer – Configurator Yes Yes Yes No No
Developer Yes Yes No No No

* Batch Jobs must be scheduled from the UI, for more details, see How to Schedule a Batch Job.

The LeadSquared admin user must generate the token for each access level from the Manage Batch Jobs page on the LeadSquared application –

  1. Login to your LeadSquared account.
  2. On the main menu, navigate to Apps>Manage Batch Jobs.
  3. On the panel on the right, under Global, click SDK Developer Token.
  4. Select the Access Level, and enter a user name (max 100 characters, only permitted special characters dot “.”,comma “,” and hyphen”-“).
  5. Click the Generate SDK Token alongside each access level. Click the view icon icon to view the token and click copy icon to copy the token.

LeadSquared Batch Jobs SDK

Note: The generated token is valid for only 30 days. Once expired, the admin must generate a new token.

 

Installation

Open a command prompt and run the following command to install the lsbatch library –

pip install lsbatch

Note: You must have Python 3.7 and virtualenv installed.

 

Initialize a New Batch Job Project

To create a new Batch Job in your local machine, navigate to your project folder and execute the following command in cmd or shell:

lsbatch init

Now enter your Batch Job name (e.g., first-batchjob) –

> Your Batch Job Name:

A new folder with the provided name will be created in the current directory. This folder contains all the libraries required for Batch Jobs local development. You will find custom virtualenv in the batch-virtualenv folder with the following default packages installed –

  • boto3==1.14.33
  • mysql-connector-python==8.0.21
  • pysftp==0.2.9
  • sendgrid==6.4.4
  • pandas==1.1.0
  • pyyaml==5.3.1
  • openpyxl==3.0.4
  • requests==2.24.0
  • tabulate==0.8.7
  • numpy==1.19.1

Note: After initiating batch with lsbatch init , make sure all the following commands are executed at the project folder level –

cd {batch_job_folder_name}

 

General Rules for Writing Batch Jobs

  1. Your Batch Job folder should have a folder named src that will be used to write your actual code. This folder is automatically created during the init process.
  2. You should keep Batch Job files in the src folder only.
  3. src folder should have main.py file that is the starting point of your application. By default, this file is created with sample code during the init process
  4. Your src folder may contain a requirements.txt file which can be used to provide any custom packages which you may require.

 

Using This Library

Batch Job provides various functions for common tasks like logging, sending notification emails, or making DB queries. lsbatch packs and mocks these functions –

 

Write Logs

Examples for writing logs are also included as comments in the sample code.

Writing a simple log –

LS.log.info('Hello')

Passing an additional JSON object (e.g., support email address) to a log –

LS.log.info('Hello', {"Support Email": "support@leadsquared.com"}

Accessing a variable value – LS.settings.<KeyName> (Key-Pair eg. Key = mykey) –

LS.log.info(LS.settings.mykey)

Writing a debug log –

LS.log.debug('Debug Log Example')

Errors and exceptions –

LS.log.error(type(err).__name__, traceback.format_exc())

LS.log.fatal(type(err).__name__, traceback.format_exc())

 

Use Variables

Used to store key-value pair with sensitive/private data. For example API key for some third-party service or your SFTP password. It can be retrieved using the following key –

LS.settings.<key_name>

 

Send Notification Emails

Automated notification emails with the default template can be configured directly from the UI, see the settings section of Create a Batch Job.

To create your own custom email content, use –

LS.email.send_email_notification(<EmailType>, <Subject>, <EmailBody>, isHTML =<True/False>, attachments =<ListofAttachments>)

Where

  • EmailType one of “Success” or “Failure” which corresponds to success or failure of execution status
  • Subject can be maximum 255 character, above which it will be trimmed
  • EmailBody is plain text or HTML content
  • isHTML is an optional parameter with default value as True
  • attachments is a list of attachments that can be passed as file names. These files are expected to be present in the user folder of Batch Jobs executions.

Note: This function can only be called max 3 times in one Batch Job execution.

 

Connect to Database

With Batch Jobs, you can directly communicate with the LeadSquared database or the LeadSquared ETL database for your account. You can use this feature to create your own customized reports.

Note: Connecting to Database is by default disabled in Batch Jobs. If you require it, please connect us at support@leadsquared.com. If it’s disabled, “Test Queries” (shown below) won’t be visible.

Use the following function to query your account database or ETL database-

LS.db.execute_DB_query(<query>, multi=False)
LS.db.execute_ETL_query(<query>, multi=False)

By default, the function considers the query as a single statement. For multiline query, pass multi parameter as True

To test the database connection and queries, under Tools, click Test Queries.

test queries

Select the database as either Tenant DB or Tenant ETL DB. Enter a query, then click Run Query. The results will display a sample of only 10 rows.

test queries

Mocking DB query result
To mock the DB result use the file query.json which is provided in the root project folder. It contains JSON where

  • key is the DB query(exact – case sensitive)
  • value is the csv file name that contains the result.

You will also find a sample_query_result.csv in project’s root folder which is mapped as a sample response in query.json.

You can create a CSV file by any name. Please note that this is purely for mocking purposes and does not evaluate the query, syntax, or result in any way. Use the admin panel in your LeadSquared account to validate the same.

 

Execute a Batch Job

To run a batch job, enter the following command –

lsbatch run

This will execute the code in context of virtual environment that has already been setup.

 

Install LeadSquared Batch Job Dependencies in the Current Folder

The root folder already contains a .gitignore file that will commit only necessary files to your source code repository. Feel free to update the file as per your need. If you clone the repository, run the lsbatch install command inside the project directory to set up the Batch Job environment again.

lsbatch install

This command can also be used in existing projects to reset the Batch Jobs environment at any time.

Note: To run this command successfully, the current directory should have src that contains your code.

 

Install Custom Packages

lsbatch install {package_name}

This installs your custom package to batch-virtualenv folder. Also it creates/updates a requirements.txt file inside src folder. requirements.txt file will be used to identify dependencies at the time of actual execution of the Batch Job.

 

Pack Your Project for Deployment

Batch Jobs accepts the deliverables as a zip file. To zip the project, execute the following command –

lsbatch pack

This will create a code.zip file in the directory named deliverables. Remember to add this directory to the .gitignore file in order to reduce the size of the repository.

 

Configure Tenant Profile

You can create multiple tenant profiles with various access levels using the configure command –

lsbatch configure

You will be prompted to enter

  • Profile Name – Define the profile name. We suggest you enter a profile name in the following convention so it is easy to remember and re-use – <orgcode>+<developer access level> (e.g., 1234Admin)
  • Orgcode – LeadSquared organization code (e.g., 1234). To find your orgcode, see How to Find My LeadSquared Orgcode.
  • Region – You may pass 1, 11, 21, 31. Alternatively, you can pass SGP, US, MUM, IR
    1 denotes Singapore | 11 denotes U.S.A | 21 denotes India | 31 denotes Ireland.
  • Token – Pass the token generated from the Batch Jobs UI.

You can also simply run lsbatch, it will prompt all available commands –

lsbatch

 

Create and Upload a Batch Job Code for Deployment

To learn how to upload the batch job code directly from the LeadSquared UI, see Create a Batch Job.

Pass the command line parameters --orgcode --region --token. You may also pass these parameters as -o, -r and -t.

E.g., lsbatch save --orgcode 1234 --region 1 --token asdfghijkl1234

Alternatively, if you’ve configured a profile use --profile or -p.

E.g., lsbatch save --profile mytenant1234

Note:

  • If the Batch Job Id doesn’t exist in the batch_details.json file, the command creates a Batch Job and uploads the code.
  • If the Batch Job Id exists in batch_detals.json, it packs and uploads the code to the Batch Job.

To save the settings, pass command line parameter --settings, or -s.

E.g., lsbatch save --profile mytenant1234 --settings

Note: Only Admins and Developer – Configurator can save the settings.

 

Deploy or Publish a Batch Job

The following command publishes a Batch Job.

Pass the command line parameters --orgcode --region --token. You may also pass these parameters as -o, -r and -t.

E.g., lsbatch publish --orgcode 1234 --region 1 --token asdfghijkl1234

Alternatively, if you’ve configured a profile use --profile or -p.

E.g., lsbatch publish --profile mytenant1234

Note: Only Admins can publish a Batch Job.

 

Uninstall Package

To uninstall a custom package, execute the following command

lsbatch uninstall {package_name}

This will also remove the package name from requirements.txt file.

 

Next Steps

Learn how to Create and Schedule a Batch Job.