| Name | SNulk JSON |
| Version |
1.0.1
JSON |
| download |
| home_page | None |
| Summary | SNulk provides a means to submit a large number of templated records to a table on a ServiceNow instance based on provided data. The table, instance, template, and data are all customizable and intended to be modified to fit the situation. |
| upload_time | 2024-10-23 16:16:00 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.11 |
| license | MIT License Copyright (c) 2024 ServiceNow, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
snulk
servicenow
pysnc
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# SNulk: ServiceNow Bulk Submit Tool for Table Records
SNulk provides a means to submit a large number of templated records to a table on a ServiceNow instance based on provided data. The table, instance, template, and data are all customizable and intended to be modified to fit the situation. A blog post illustrating how to use SNulk is coming soon. Check the [Security Research Blog Posts](https://securitylab.servicenow.com/research/) for updates.
## Setup
Below are instructions on how to setup and install SNulk and its dependencies.
### Install Dependencies
SNulk is a python CLI tool that uses [rye](https://rye.astral.sh/guide/installation/#installing-rye), [selenium](https://pypi.org/project/selenium/), and firefox. You will also likely need to install [geckodriver](https://github.com/mozilla/geckodriver/releases) so selenium can communicate with firefox. Note selenium, firefox, and geckodriver are only used to authenticate with an instance when not using basic auth credentials. If you plan to only use basic auth credentials for authentication then these dependencies are not required.
Detailed instructions for installing the dependencies can be found on their website. Below are general commands on how one might install these dependencies for MacOS.
```bash
# Install rye
brew install rye
echo 'source "$HOME/.rye/env"' >> ~/.zprofile
echo 'source "$HOME/.rye/env"' >> ~/.zshrc
# Install firefox
brew install --cask firefox
# Install selenium (not needed if snulk is only being run inside the venv)
pip install selenium
# Install geckodriver
brew install geckodriver
```
### Install SNulk
We provide three ways to install and run SNulk: from PyPi, from source inside a virtual environment, and built and installed from source. The commands below detail these three install methods.
#### From PyPi (Recommended)
```bash
pip install SNulk
# See below for more cli command examples and options
snulk -h
# If you wish to uninstall snulk run this command
pip uninstall SNulk
```
#### From Source - In A Virtual Environment
```bash
git clone https://github.com/ServiceNow/SNulk.git
rye sync
# See below for more cli command examples and options
rye run snulk -h
```
#### From Source - Build And Install
```bash
git clone https://github.com/ServiceNow/SNulk.git
rye sync
rye build --clean
pip install --break-system-packages --user ./dist/snulk-*.whl
# See below for more cli command examples and options
python -m snulk -h
# If you wish to uninstall snulk run this command
pip uninstall snulk
```
## Running SNulk
SNulk can be used as both a CLI tool and as a python library by other tools. The examples below illustrate these two uses.
### SNulk CLI
The SNulk CLI is the easiest way to use SNulk. Below are some examples of how to use the CLI and full details of the CLI options. Note the input files mentioned are explained in a later section.
#### CLI Command Examples
The commands below assume SNulk was installed from PyPi. If SNulk was installed using a different method, replace `snulk` with either `rye run snulk` for the virtual environment or `python -m snulk` for the build from source method.
```bash
# Uses basic auth to authenticate to an instance
snulk -t submit_table -u user -p pass -n short_name -i data/file.xlsx > log.txt
# Uses selenium and firefox to manually login to and instance and grab the cookies
# You may need to manually navigate to /now/nav/ui/classic/params/target/ after login
snulk -t submit_table -n short_name -i data/file.xlsx > log.txt
# Only read submit data from sheet1 and sheet2 of file.xlsx
snulk -t submit_table -n short_name -i data/file.xlsx sheet1 sheet2 > log.txt
```
#### CLI Usage Instructions
```bash
usage: SNulk: ServiceNow Bulk Submit Tool for Table Records [-h] [--table_dir STD] [--format_table STD_FORMAT]
[--struct_table STD_STRUCT] [--username USERNAME]
[--password PASSWORD] --name SHORT_NAME --input INPUT
[INPUT ...] [--debug]
This tool provides a means to submit a large number of templated records to a table on a ServiceNow instance based on
provided data. The table, instance, template, and data are all customizable and intended to be modified to fit the
situation.
options:
-h, --help show this help message and exit
--table_dir STD, -t STD The directory containing yaml files that define the structures of a
tables and the formats (templates) used to map input data to table
fields. The directory should contain two sub directories with the
names 'format' and 'struct' that house the associated yaml files.
This option can be given multiple times. If this option,
--format_table, or --struct_table is not given, SNulk will default
to using --table_dir '<cwd>/submit_table'.
--format_table STD_FORMAT, -f STD_FORMAT A path to either a format table yaml file or a directory containing
format table yaml files. This option can be given multiple files.
If this option, --table_dir, or --struct_table is not given, SNulk
will default to using --table_dir '<cwd>/submit_table'.
--struct_table STD_STRUCT, -s STD_STRUCT A path to either a struct table yaml file or a directory containing
struct table yaml files. This option can be given multiple files.
If this option, --table_dir, or --format_table is not given, SNulk
will default to using --table_dir '<cwd>/submit_table'.
--username USERNAME, -u USERNAME The username to be used to login to an instance using basic auth.
If given then password must also be specified. If neither username
or password is given then authorization is conducted using the
session data obtained using selenium and firefox.
--password PASSWORD, -p PASSWORD The password to be used to login to an instance using basic auth.
If given then username must also be specified. If neither username
or password is given the authorization is conducted using the
session data obtained using selenium and firefox.
--name SHORT_NAME, -n SHORT_NAME The name of a format to be used when submitting data. This should
be the same as the 'short_name' value of one of the yaml files
provided as a format.
--input INPUT [INPUT ...], -i INPUT [INPUT ...] The input data. Currently only xlsx files are supported. The value
is a list where the first element is an xlsx file and the remaining
are sheet names in the file to be used when submitting data. For
example, '-i file.xlsx s1 s2 s3' will read from sheets s1, s2, and
s3 of file.xlsx when submitting records to the specified table. If
file.xlsx contained any additional sheets those sheets would be
ignored. If just '-i file.xlsx' is given then all sheets will be
used. The '-i' argument may be given multiple times to provide
multiple data sources.
--debug, -d Enable debug logging output.
```
### SNulk Library
SNulk can also be used as a library to provide further customization or use with other scripts. Below is a example on how to use SNulk as a library using the example data found under the `example` directory. This example illustrates multiple ways to perform the same task. Pick and choose what best fits your use case.
```python
from snulk import BulkSubmitter
from pathlib import Path
bs = BulkSubmitter()
# load the struct and format files
# make sure all struct files are loaded before format files
# load a single struct yaml file
bs.load_struct_submit_tables_from_file(Path("example/submit_table/struct/dev_incident.yaml"))
# load a single format yaml file
bs.load_format_submit_tables_from_file(Path("example/submit_table/format/test_submit.yaml"))
# load an entire directory of struct and format yaml files
bs.load_submit_tables_from_dir(Path("example/submit_table"))
# load input data
bs.load_data_file(Path("example/data/test_submit.xlsx"))
# submit input data using basic auth
bs.bulk_submit_all("test_incident", "username", "password")
# you can also use captured sessions to login
# you may need to navigate to '/now/nav/ui/classic/params/target/' manually
bs.bulk_submit_all("test_incident")
# it is also possible to use your own data frame for submission
bs.bulk_submit_basicauth("test_incident", data, "username", "password", Path("example/data/out.xlsx"), "Sheet_name")
# same as the previous example but using captured sessions to login
# you may need to navigate to '/now/nav/ui/classic/params/target/' manually
bs.bulk_submit_session("test_incident", data, Path("example/data/out.xlsx"), "Sheet_name")
```
## Input and Output Files
### Data Files
The required `--input` option takes as input a file and a list of sheet names. Currently, only xlsx files are supported. All xlsx files given are considered input sources for SNulk and will be used for submission. If sheet names are given, only the the sheet names listed for each file will be used for submission. Otherwise all sheets in a file will be used for submission. See the above help message for more information on the formatting and how to use the input option.
While only some input file sheet names may be used for submission, all input file sheets are loaded by Snulk. Snulk uses `pandas` to read and write input data. As such, all sheets of every file given as input are loaded as `Dataframes` and grouped by input file path and sheet name.
Any data that is requested to be retrieved after a successful submission is written back to the associated `Dataframe` from which the submission data was retrieved. Data is not written back to the originating file until all rows in a `Dataframe` have been submitted. If multiple `Dataframes` originate from the same file (i.e. the file had multiple sheets), all `Dataframes` will be written to the file on any given write. This is possible because all sheets have been loaded as `Dataframes` even if they are not being used. Upon some exception during the submission process of a `Dataframe`, any data successfully retrieved already and stored in the `Dataframe` is written back to its originating file before SNulk exits. This process for syncing `Dataframes` and files should preserve any unused sheets and data while ensuring any new data is recorded baring some problem with the file system itself.
Note that any `Dataframe` used for submission will have at least one additional column added to it. This column stores the `sysid` of the submitted record. This allows SNulk to be run multiple times on the same input data and format file without duplicating submissions. That is if a `Dataframe` has a `sysid` already recorded for a given row, that row will be skipped with processing the `Dataframe`. The name of the column that stores the `sysid` of submitted records can be customized by specifying a **Format Return Field** with the special name `__SYSID__`. If a **Format Return Field** is not given with this special name then the column storing `sysid` defaults to the name `Submit SysId`. For more information on how to specify the data retrieved after a submission see the **Format Return Fields** section.
### Yaml Config Files
The required `--table_dir` option takes as input a directory that must contain the folders `format` and `struct`. These folders should contains the available **Format Yaml Files** and **Struct Yaml Files** respectively. The specification for these files is outlined below. All yaml files in these folders will be loaded and be available to SNulk for use regardless of what `--name` option is given.
Note **Struct Yaml Files** are always loaded first to provide the basic structure of the available tables and fields to SNulk. **Format Yaml Files** are then loaded to provide SNulk specific information about what fields and data will actually be submitted for a specifically named (`short_name`) format. SNulk will produce warnings about fields and tables not defined in `struct` files but used in `format` files but will not stop processing. It will also warn about duplicate `struct` (same `instance` and `table` elements) or `format` (same `short_name`) files.
### Format Yaml Files
Below is an example of a format (template) yaml file that is used to map fields in the provided input data with the fields of a table on a specific instance.
```yml
---
short_name: name_of_this_format
instance: instance_name
table: table_name
return_fields:
- name: __SYSID__
data_key: new_field_name_in_data_1
none_is_empty: true
- name: number
data_key: new_field_name_in_data_2
none_is_empty: true
fields:
- name: field_name_in_table
data_key: field_name_in_data
default_value: "A example default value '[!--some_field_in_data--!]'"
required: true
substitution: true
append_hash: true
empty_is_none: true
```
#### Format Header
Each format structure starts with a header that must contain the following entries.
```yml
short_name: name_of_this_format
instance: instance_name
table: table_name
return_fields:
fields:
```
##### short_name - (required) - (`[0-9a-zA-Z\-_]+`)
<sub>The name used to identify this format. This is what gets provided to the CLI under the option `--name`.</sub>
##### instance - (required) - (https url or `[0-9a-zA-Z\-_]+`)
<sub>The name of an instance (i.e. from `<instance-name>.service-now.com` or the full https url of an instance). This will be the instance that SNulk connects to in order to write data.</sub>
##### table_name - (required) - (`[0-9a-zA-Z\-_]+`)
<sub>The name of a table on the specified instance. This will be the table that is written to.</sub>
##### return_fields - (required) - (list containing 0 or more entries of type return_field)
<sub>This is a list that specifies the format options for return_fields. A return_field is a field of a table that SNulk retrieves after the new record has been submitted. Return fields map table data back into our input data to record information that gets generated server side once a record is created. This list can be empty but the entry for it must still be specified. For more information see **Format Return Fields** below.</sub>
##### fields - (required) - (list containing 1 or more entries of type field)
<sub>This is a list that specified the format options for fields. A field is a field of a table that SNulk includes when submitting a new record. Fields map input data to table data. This list cannot be empty. For more information see **Format Fields** below.</sub>
#### Format Return Fields
Each format structure can contain 0 or more return fields as specified below. These fields map data from the table back to our input data once a record has been submitted.
```yml
return_fields:
- name: __SYSID__
data_key: new_field_name_in_data_1
none_is_empty: true
- name: number
data_key: new_field_name_in_data_2
none_is_empty: true
```
##### name - (required) - (`[0-9a-zA-Z\-_]+`)
<sub>Name of a field in the table that SNulk will be reading data from once a record is submitted. To specify that the sysid of the submitted record should be retrieved, use the special name `__SYSID__` as shown above.</sub>
##### data_key - (required) - (`[0-9a-zA-Z\-_]+`)
<sub>The name of a field in the input data that SNulk will be writing data to once a record is submitted. The data is retrieved from field `name` of the table.</sub>
##### none_is_empty - (optional: default = `true`) - (`true/false`)
<sub>Indicates if `None`, `null`, or other data types that `pandas` considers null data (e.g. `NA`) should be written as an empty string when writing to the input data.</sub>
#### Format Fields
Format fields map the values of fields in input data to the fields of a ServiceNow table. The format field and its elements are outlined below.
Note the order of evaluation when determining a value for field `name` of the table record is: 1) `data_key`, 2) `default_value`. An element is determined to not be a valid value for field `name` if its value is a null value as determined by `pandas` or if `empty_is_none=true` and the elements value is an empty string. If no valid value is found an exception will be thrown unless `required=false`.
```yml
fields:
- name: field_name_in_table
data_key: field_name_in_data
default_value: "A example default value '[!--some_field_in_data--!]'"
required: true
substitution: true
append_hash: true
empty_is_none: true
```
##### name - (required) - (`[0-9a-zA-Z\-_]+`)
<sub>Name of a field in the table that SNulk will be writing data to.</sub>
##### data_key - (optional: default = `None`) - (non-empty string)
<sub>The name of a field in the input data that SNulk will retrieve data from to write to field `name` in the table. This element of a format field does not need to be specified as data can also come from the `default_value` element.</sub>
##### default_value - (optional: default = `None`) - (non-empty string)
<sub>A non-empty string of some value that will be used as the value for field `name` if `data_key` is not given or if the row from input data have no value for `data_key`. This element of a format field does not need to be specified.</sub>
<sub>Note to specify a multi-line string in yaml use the format below:</sub>
```yml
fields:
- name: engineering_details
default_value: |
line1
line2
line3
required: true
```
##### required - (optional: default = `false`) - (true/false)
<sub>Specifies if a value for the field `name` must be determined before submitting a record to the table. If false, the field `name` will be omitted from the submitted record if no value can be determined. If true and no value can be determined, a exception will be thrown.</sub>
##### substitution - (optional: default = `false`) - (true/false)
<sub>Specifies if once a value for field `name` is determined, substitution should be performed. Substitution allows for the values of different fields of the same row from input data to be combined into a single value for submission to a table. The substitution signature is as follows `[!--input_field_name--!]`. Substitution can be used on the values retrieved from `data_key` or `default_value`. There can be multiple substitutions in the same string. Empty strings being none are controlled by the same `empty_is_none` false described below. This is if `empty_is_none=true` and the value for `input_field_name` of the input data is an empty string, it is treated as none. Attempting to substitute any null value, as determined by `pandas`, will raise an exception.</sub>
<sub>Example substitution in `default_value`:</sub>
```yml
fields:
- name: engineering_details
default_value: |
Some text
[!--input_field_name--!]
Some more text [!--input_field_name_2--!]
required: true
```
##### append_hash - (optional: default = `false`) - (true/false)
<sub>Once all the values for a new record are determined, SNulk will hash those values and append the hash to any field where `append_hash=true`. Before submission, SNulk will query the table to find any record whose field (i.e. the one with `append_hash`) contains the hash of the data about to be submitted. If an existing record exists, SNulk warns the user that a duplicate is about to be submitted and asks if they wish to continue. This is a means to prevent duplicate submissions. The append value will be of the form `Automated submission by BulkSubmitter - SHA256:[hash]`. If you don't wish to have hashes appended or don't wish to perform this check, just remove any `append_hash` elements.</sub>
##### empty_is_none - (optional: default = `true`) - (true/false)
<sub>When set to true empty strings are treated as null values. That is, once a value is determined and it is an empty string, `empty_is_none=true` will cause the value to be treated as null. As null values as determined by `pandas` are errors, an exception will be raised. Note this element will also effect the behavior of substitution.</sub>
### Struct Yaml File
Below is an example of a struct (table structure) yaml file. These files are used to define the structure of fields in a table on a specific instance. Specifically, they allow the user to define available fields of a table and possible values for those fields. Currently this is used to alert the user when they are accessing a field not defined in a table or trying to assign a invalid value to a field. They can also be referenced by user to figure out what fields are available and what values are possible for those fields. The information is currently manually defined but future efforts will attempt to automate the building of these files.
If a struct file is not defined for a given table and instance combination, it will produce warnings but not stop SNulk functionality. If you wish to get rid of the warnings simply define a struct file for a give table and instance combination and list the names of all the fields used in the associated format file.
```yml
---
instance: instance_name
table: table_name
fields:
- name: field_name_in_table
- name: field_name_in_table_2
possible_values:
- id: 'id1'
short_description: 'Des1'
- id: 'id2'
short_description: 'des2'
- id: 'id3'
short_description: 'des3'
```
#### Struct Header
Each table structure starts with a header that must contain the following entries.
```yml
instance: instance_name
table: table_name
fields:
```
##### instance - (required) - (`https url or [0-9a-zA-Z\-_]+`)
<sub>The name of an instance (i.e. from `<instance-name>.service-now.com` or the full https url of an instance). This will be the instance the table is on. It should correspond to a table/instance pair in the format yaml files to be used.</sub>
##### table_name - (required) - (`[0-9a-zA-Z\-_]+`)
<sub>The name of a table on the specified instance. This will be the table from an instance. It should correspond to a table/instance pair in the format yaml files to be used.</sub>
##### fields - (required) - (list containing 1 or more entries of type field)
<sub>This is a list that specifies the available fields for a table on a given instance. This list cannot be empty. For more information see **Struct Fields** below.</sub>
#### Struct Fields
Struct fields are used to define the available fields for a table on an instance. The elements of struct fields are outlined below.
```yml
fields:
- name: field_name_in_table
- name: field_name_in_table_2
possible_values:
- id: 'id1'
short_description: 'Des1'
- id: 'id2'
short_description: 'des2'
- id: 'id3'
short_description: 'des3'
```
##### name - (required) - (`[0-9a-zA-Z\-_]+`)
<sub>Name of a field on the table on an instance.</sub>
##### possible_values - (optional: default = `None`) - (list containing 1 or more entries of type field)
<sub>A list of possible values for the field with a given name. A possible value contains two elements: `id` and `short_description`. The `id` is the value that will actually be submitted to an instance while the `short_description` is a more human readable string that describes what the value means. Specifying the list of `possible_values` is not required but if specified the list must contain at leas one element.</sub>
Raw data
{
"_id": null,
"home_page": null,
"name": "SNulk",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": "\"sigmund.gorski\" <sigmund.gorski@servicenow.com>",
"keywords": "SNulk, ServiceNow, pysnc",
"author": null,
"author_email": "\"sigmund.gorski\" <sigmund.gorski@servicenow.com>",
"download_url": "https://files.pythonhosted.org/packages/7f/89/aaf9efdf12cd0b85d9188622c2735a5cb030b7c27ef801a1493217ddc997/snulk-1.0.1.tar.gz",
"platform": null,
"description": "# SNulk: ServiceNow Bulk Submit Tool for Table Records\n\nSNulk provides a means to submit a large number of templated records to a table on a ServiceNow instance based on provided data. The table, instance, template, and data are all customizable and intended to be modified to fit the situation. A blog post illustrating how to use SNulk is coming soon. Check the [Security Research Blog Posts](https://securitylab.servicenow.com/research/) for updates.\n\n## Setup\n\nBelow are instructions on how to setup and install SNulk and its dependencies.\n\n### Install Dependencies\n\nSNulk is a python CLI tool that uses [rye](https://rye.astral.sh/guide/installation/#installing-rye), [selenium](https://pypi.org/project/selenium/), and firefox. You will also likely need to install [geckodriver](https://github.com/mozilla/geckodriver/releases) so selenium can communicate with firefox. Note selenium, firefox, and geckodriver are only used to authenticate with an instance when not using basic auth credentials. If you plan to only use basic auth credentials for authentication then these dependencies are not required. \n\nDetailed instructions for installing the dependencies can be found on their website. Below are general commands on how one might install these dependencies for MacOS.\n\n```bash\n# Install rye\nbrew install rye\necho 'source \"$HOME/.rye/env\"' >> ~/.zprofile\necho 'source \"$HOME/.rye/env\"' >> ~/.zshrc\n\n# Install firefox\nbrew install --cask firefox\n\n# Install selenium (not needed if snulk is only being run inside the venv)\npip install selenium\n\n# Install geckodriver\nbrew install geckodriver\n```\n\n### Install SNulk\n\nWe provide three ways to install and run SNulk: from PyPi, from source inside a virtual environment, and built and installed from source. The commands below detail these three install methods.\n\n#### From PyPi (Recommended)\n\n```bash\npip install SNulk\n# See below for more cli command examples and options\nsnulk -h\n# If you wish to uninstall snulk run this command\npip uninstall SNulk\n```\n\n#### From Source - In A Virtual Environment\n\n```bash\ngit clone https://github.com/ServiceNow/SNulk.git\nrye sync\n# See below for more cli command examples and options\nrye run snulk -h\n```\n\n#### From Source - Build And Install\n\n```bash\ngit clone https://github.com/ServiceNow/SNulk.git\nrye sync\nrye build --clean\npip install --break-system-packages --user ./dist/snulk-*.whl\n# See below for more cli command examples and options\npython -m snulk -h\n# If you wish to uninstall snulk run this command\npip uninstall snulk\n```\n\n## Running SNulk\n\nSNulk can be used as both a CLI tool and as a python library by other tools. The examples below illustrate these two uses.\n\n### SNulk CLI\n\nThe SNulk CLI is the easiest way to use SNulk. Below are some examples of how to use the CLI and full details of the CLI options. Note the input files mentioned are explained in a later section.\n\n#### CLI Command Examples\n\nThe commands below assume SNulk was installed from PyPi. If SNulk was installed using a different method, replace `snulk` with either `rye run snulk` for the virtual environment or `python -m snulk` for the build from source method.\n\n```bash\n# Uses basic auth to authenticate to an instance\nsnulk -t submit_table -u user -p pass -n short_name -i data/file.xlsx > log.txt\n\n# Uses selenium and firefox to manually login to and instance and grab the cookies\n# You may need to manually navigate to /now/nav/ui/classic/params/target/ after login\nsnulk -t submit_table -n short_name -i data/file.xlsx > log.txt\n\n# Only read submit data from sheet1 and sheet2 of file.xlsx\nsnulk -t submit_table -n short_name -i data/file.xlsx sheet1 sheet2 > log.txt\n```\n\n#### CLI Usage Instructions\n\n```bash\nusage: SNulk: ServiceNow Bulk Submit Tool for Table Records [-h] [--table_dir STD] [--format_table STD_FORMAT]\n [--struct_table STD_STRUCT] [--username USERNAME]\n [--password PASSWORD] --name SHORT_NAME --input INPUT\n [INPUT ...] [--debug]\n\nThis tool provides a means to submit a large number of templated records to a table on a ServiceNow instance based on\nprovided data. The table, instance, template, and data are all customizable and intended to be modified to fit the\nsituation.\n\noptions:\n -h, --help show this help message and exit\n --table_dir STD, -t STD The directory containing yaml files that define the structures of a\n tables and the formats (templates) used to map input data to table\n fields. The directory should contain two sub directories with the\n names 'format' and 'struct' that house the associated yaml files.\n This option can be given multiple times. If this option,\n --format_table, or --struct_table is not given, SNulk will default\n to using --table_dir '<cwd>/submit_table'.\n --format_table STD_FORMAT, -f STD_FORMAT A path to either a format table yaml file or a directory containing\n format table yaml files. This option can be given multiple files.\n If this option, --table_dir, or --struct_table is not given, SNulk\n will default to using --table_dir '<cwd>/submit_table'.\n --struct_table STD_STRUCT, -s STD_STRUCT A path to either a struct table yaml file or a directory containing\n struct table yaml files. This option can be given multiple files.\n If this option, --table_dir, or --format_table is not given, SNulk\n will default to using --table_dir '<cwd>/submit_table'.\n --username USERNAME, -u USERNAME The username to be used to login to an instance using basic auth.\n If given then password must also be specified. If neither username\n or password is given then authorization is conducted using the\n session data obtained using selenium and firefox.\n --password PASSWORD, -p PASSWORD The password to be used to login to an instance using basic auth.\n If given then username must also be specified. If neither username\n or password is given the authorization is conducted using the\n session data obtained using selenium and firefox.\n --name SHORT_NAME, -n SHORT_NAME The name of a format to be used when submitting data. This should\n be the same as the 'short_name' value of one of the yaml files\n provided as a format.\n --input INPUT [INPUT ...], -i INPUT [INPUT ...] The input data. Currently only xlsx files are supported. The value\n is a list where the first element is an xlsx file and the remaining\n are sheet names in the file to be used when submitting data. For\n example, '-i file.xlsx s1 s2 s3' will read from sheets s1, s2, and\n s3 of file.xlsx when submitting records to the specified table. If\n file.xlsx contained any additional sheets those sheets would be\n ignored. If just '-i file.xlsx' is given then all sheets will be\n used. The '-i' argument may be given multiple times to provide\n multiple data sources.\n --debug, -d Enable debug logging output.\n```\n\n### SNulk Library\n\nSNulk can also be used as a library to provide further customization or use with other scripts. Below is a example on how to use SNulk as a library using the example data found under the `example` directory. This example illustrates multiple ways to perform the same task. Pick and choose what best fits your use case.\n\n```python\nfrom snulk import BulkSubmitter\nfrom pathlib import Path\n\nbs = BulkSubmitter()\n\n# load the struct and format files\n# make sure all struct files are loaded before format files\n# load a single struct yaml file\nbs.load_struct_submit_tables_from_file(Path(\"example/submit_table/struct/dev_incident.yaml\"))\n# load a single format yaml file\nbs.load_format_submit_tables_from_file(Path(\"example/submit_table/format/test_submit.yaml\"))\n# load an entire directory of struct and format yaml files\nbs.load_submit_tables_from_dir(Path(\"example/submit_table\"))\n\n# load input data\nbs.load_data_file(Path(\"example/data/test_submit.xlsx\"))\n# submit input data using basic auth\nbs.bulk_submit_all(\"test_incident\", \"username\", \"password\")\n\n# you can also use captured sessions to login\n# you may need to navigate to '/now/nav/ui/classic/params/target/' manually\nbs.bulk_submit_all(\"test_incident\")\n\n# it is also possible to use your own data frame for submission\nbs.bulk_submit_basicauth(\"test_incident\", data, \"username\", \"password\", Path(\"example/data/out.xlsx\"), \"Sheet_name\")\n\n# same as the previous example but using captured sessions to login\n# you may need to navigate to '/now/nav/ui/classic/params/target/' manually\nbs.bulk_submit_session(\"test_incident\", data, Path(\"example/data/out.xlsx\"), \"Sheet_name\")\n```\n\n## Input and Output Files\n\n### Data Files\n\nThe required `--input` option takes as input a file and a list of sheet names. Currently, only xlsx files are supported. All xlsx files given are considered input sources for SNulk and will be used for submission. If sheet names are given, only the the sheet names listed for each file will be used for submission. Otherwise all sheets in a file will be used for submission. See the above help message for more information on the formatting and how to use the input option.\n\nWhile only some input file sheet names may be used for submission, all input file sheets are loaded by Snulk. Snulk uses `pandas` to read and write input data. As such, all sheets of every file given as input are loaded as `Dataframes` and grouped by input file path and sheet name.\n\nAny data that is requested to be retrieved after a successful submission is written back to the associated `Dataframe` from which the submission data was retrieved. Data is not written back to the originating file until all rows in a `Dataframe` have been submitted. If multiple `Dataframes` originate from the same file (i.e. the file had multiple sheets), all `Dataframes` will be written to the file on any given write. This is possible because all sheets have been loaded as `Dataframes` even if they are not being used. Upon some exception during the submission process of a `Dataframe`, any data successfully retrieved already and stored in the `Dataframe` is written back to its originating file before SNulk exits. This process for syncing `Dataframes` and files should preserve any unused sheets and data while ensuring any new data is recorded baring some problem with the file system itself. \n\nNote that any `Dataframe` used for submission will have at least one additional column added to it. This column stores the `sysid` of the submitted record. This allows SNulk to be run multiple times on the same input data and format file without duplicating submissions. That is if a `Dataframe` has a `sysid` already recorded for a given row, that row will be skipped with processing the `Dataframe`. The name of the column that stores the `sysid` of submitted records can be customized by specifying a **Format Return Field** with the special name `__SYSID__`. If a **Format Return Field** is not given with this special name then the column storing `sysid` defaults to the name `Submit SysId`. For more information on how to specify the data retrieved after a submission see the **Format Return Fields** section.\n\n### Yaml Config Files\n\nThe required `--table_dir` option takes as input a directory that must contain the folders `format` and `struct`. These folders should contains the available **Format Yaml Files** and **Struct Yaml Files** respectively. The specification for these files is outlined below. All yaml files in these folders will be loaded and be available to SNulk for use regardless of what `--name` option is given. \n\nNote **Struct Yaml Files** are always loaded first to provide the basic structure of the available tables and fields to SNulk. **Format Yaml Files** are then loaded to provide SNulk specific information about what fields and data will actually be submitted for a specifically named (`short_name`) format. SNulk will produce warnings about fields and tables not defined in `struct` files but used in `format` files but will not stop processing. It will also warn about duplicate `struct` (same `instance` and `table` elements) or `format` (same `short_name`) files.\n\n### Format Yaml Files\n\nBelow is an example of a format (template) yaml file that is used to map fields in the provided input data with the fields of a table on a specific instance.\n\n```yml\n---\nshort_name: name_of_this_format\ninstance: instance_name\ntable: table_name\nreturn_fields:\n - name: __SYSID__\n data_key: new_field_name_in_data_1\n none_is_empty: true\n - name: number\n data_key: new_field_name_in_data_2\n none_is_empty: true\nfields:\n - name: field_name_in_table\n data_key: field_name_in_data\n default_value: \"A example default value '[!--some_field_in_data--!]'\"\n required: true\n substitution: true\n append_hash: true\n empty_is_none: true\n```\n\n#### Format Header\n\nEach format structure starts with a header that must contain the following entries.\n\n```yml\nshort_name: name_of_this_format\ninstance: instance_name\ntable: table_name\nreturn_fields:\nfields:\n```\n\n##### short_name - (required) - (`[0-9a-zA-Z\\-_]+`)\n\n<sub>The name used to identify this format. This is what gets provided to the CLI under the option `--name`.</sub>\n\n##### instance - (required) - (https url or `[0-9a-zA-Z\\-_]+`)\n\n<sub>The name of an instance (i.e. from `<instance-name>.service-now.com` or the full https url of an instance). This will be the instance that SNulk connects to in order to write data.</sub>\n\n##### table_name - (required) - (`[0-9a-zA-Z\\-_]+`)\n\n<sub>The name of a table on the specified instance. This will be the table that is written to.</sub>\n\n##### return_fields - (required) - (list containing 0 or more entries of type return_field)\n\n<sub>This is a list that specifies the format options for return_fields. A return_field is a field of a table that SNulk retrieves after the new record has been submitted. Return fields map table data back into our input data to record information that gets generated server side once a record is created. This list can be empty but the entry for it must still be specified. For more information see **Format Return Fields** below.</sub>\n\n##### fields - (required) - (list containing 1 or more entries of type field)\n\n<sub>This is a list that specified the format options for fields. A field is a field of a table that SNulk includes when submitting a new record. Fields map input data to table data. This list cannot be empty. For more information see **Format Fields** below.</sub>\n\n#### Format Return Fields\n\nEach format structure can contain 0 or more return fields as specified below. These fields map data from the table back to our input data once a record has been submitted.\n\n```yml\nreturn_fields:\n - name: __SYSID__\n data_key: new_field_name_in_data_1\n none_is_empty: true\n - name: number\n data_key: new_field_name_in_data_2\n none_is_empty: true\n```\n\n##### name - (required) - (`[0-9a-zA-Z\\-_]+`)\n\n<sub>Name of a field in the table that SNulk will be reading data from once a record is submitted. To specify that the sysid of the submitted record should be retrieved, use the special name `__SYSID__` as shown above.</sub>\n\n##### data_key - (required) - (`[0-9a-zA-Z\\-_]+`)\n\n<sub>The name of a field in the input data that SNulk will be writing data to once a record is submitted. The data is retrieved from field `name` of the table.</sub>\n\n##### none_is_empty - (optional: default = `true`) - (`true/false`)\n\n<sub>Indicates if `None`, `null`, or other data types that `pandas` considers null data (e.g. `NA`) should be written as an empty string when writing to the input data.</sub>\n\n#### Format Fields\n\nFormat fields map the values of fields in input data to the fields of a ServiceNow table. The format field and its elements are outlined below.\n\nNote the order of evaluation when determining a value for field `name` of the table record is: 1) `data_key`, 2) `default_value`. An element is determined to not be a valid value for field `name` if its value is a null value as determined by `pandas` or if `empty_is_none=true` and the elements value is an empty string. If no valid value is found an exception will be thrown unless `required=false`.\n\n```yml\nfields:\n - name: field_name_in_table\n data_key: field_name_in_data\n default_value: \"A example default value '[!--some_field_in_data--!]'\"\n required: true\n substitution: true\n append_hash: true\n empty_is_none: true\n```\n\n##### name - (required) - (`[0-9a-zA-Z\\-_]+`)\n\n<sub>Name of a field in the table that SNulk will be writing data to.</sub>\n\n##### data_key - (optional: default = `None`) - (non-empty string)\n\n<sub>The name of a field in the input data that SNulk will retrieve data from to write to field `name` in the table. This element of a format field does not need to be specified as data can also come from the `default_value` element.</sub>\n\n##### default_value - (optional: default = `None`) - (non-empty string)\n\n<sub>A non-empty string of some value that will be used as the value for field `name` if `data_key` is not given or if the row from input data have no value for `data_key`. This element of a format field does not need to be specified.</sub>\n\n<sub>Note to specify a multi-line string in yaml use the format below:</sub>\n\n```yml\nfields:\n - name: engineering_details\n default_value: |\n line1\n line2\n line3\n required: true\n```\n\n##### required - (optional: default = `false`) - (true/false)\n\n<sub>Specifies if a value for the field `name` must be determined before submitting a record to the table. If false, the field `name` will be omitted from the submitted record if no value can be determined. If true and no value can be determined, a exception will be thrown.</sub>\n\n##### substitution - (optional: default = `false`) - (true/false)\n\n<sub>Specifies if once a value for field `name` is determined, substitution should be performed. Substitution allows for the values of different fields of the same row from input data to be combined into a single value for submission to a table. The substitution signature is as follows `[!--input_field_name--!]`. Substitution can be used on the values retrieved from `data_key` or `default_value`. There can be multiple substitutions in the same string. Empty strings being none are controlled by the same `empty_is_none` false described below. This is if `empty_is_none=true` and the value for `input_field_name` of the input data is an empty string, it is treated as none. Attempting to substitute any null value, as determined by `pandas`, will raise an exception.</sub>\n\n<sub>Example substitution in `default_value`:</sub>\n\n```yml\nfields:\n - name: engineering_details\n default_value: |\n Some text\n [!--input_field_name--!]\n Some more text [!--input_field_name_2--!]\n required: true\n```\n\n##### append_hash - (optional: default = `false`) - (true/false)\n\n<sub>Once all the values for a new record are determined, SNulk will hash those values and append the hash to any field where `append_hash=true`. Before submission, SNulk will query the table to find any record whose field (i.e. the one with `append_hash`) contains the hash of the data about to be submitted. If an existing record exists, SNulk warns the user that a duplicate is about to be submitted and asks if they wish to continue. This is a means to prevent duplicate submissions. The append value will be of the form `Automated submission by BulkSubmitter - SHA256:[hash]`. If you don't wish to have hashes appended or don't wish to perform this check, just remove any `append_hash` elements.</sub>\n\n##### empty_is_none - (optional: default = `true`) - (true/false)\n\n<sub>When set to true empty strings are treated as null values. That is, once a value is determined and it is an empty string, `empty_is_none=true` will cause the value to be treated as null. As null values as determined by `pandas` are errors, an exception will be raised. Note this element will also effect the behavior of substitution.</sub>\n\n### Struct Yaml File\n\nBelow is an example of a struct (table structure) yaml file. These files are used to define the structure of fields in a table on a specific instance. Specifically, they allow the user to define available fields of a table and possible values for those fields. Currently this is used to alert the user when they are accessing a field not defined in a table or trying to assign a invalid value to a field. They can also be referenced by user to figure out what fields are available and what values are possible for those fields. The information is currently manually defined but future efforts will attempt to automate the building of these files. \n\nIf a struct file is not defined for a given table and instance combination, it will produce warnings but not stop SNulk functionality. If you wish to get rid of the warnings simply define a struct file for a give table and instance combination and list the names of all the fields used in the associated format file.\n\n```yml\n---\ninstance: instance_name\ntable: table_name\nfields:\n - name: field_name_in_table\n - name: field_name_in_table_2\n possible_values:\n - id: 'id1'\n short_description: 'Des1'\n - id: 'id2'\n short_description: 'des2'\n - id: 'id3'\n short_description: 'des3'\n```\n\n#### Struct Header\n\nEach table structure starts with a header that must contain the following entries.\n\n```yml\ninstance: instance_name\ntable: table_name\nfields:\n```\n\n##### instance - (required) - (`https url or [0-9a-zA-Z\\-_]+`)\n\n<sub>The name of an instance (i.e. from `<instance-name>.service-now.com` or the full https url of an instance). This will be the instance the table is on. It should correspond to a table/instance pair in the format yaml files to be used.</sub>\n\n##### table_name - (required) - (`[0-9a-zA-Z\\-_]+`)\n\n<sub>The name of a table on the specified instance. This will be the table from an instance. It should correspond to a table/instance pair in the format yaml files to be used.</sub>\n\n##### fields - (required) - (list containing 1 or more entries of type field)\n\n<sub>This is a list that specifies the available fields for a table on a given instance. This list cannot be empty. For more information see **Struct Fields** below.</sub>\n\n#### Struct Fields\n\nStruct fields are used to define the available fields for a table on an instance. The elements of struct fields are outlined below.\n\n```yml\nfields:\n - name: field_name_in_table\n - name: field_name_in_table_2\n possible_values:\n - id: 'id1'\n short_description: 'Des1'\n - id: 'id2'\n short_description: 'des2'\n - id: 'id3'\n short_description: 'des3'\n```\n\n##### name - (required) - (`[0-9a-zA-Z\\-_]+`)\n\n<sub>Name of a field on the table on an instance.</sub>\n\n##### possible_values - (optional: default = `None`) - (list containing 1 or more entries of type field)\n\n<sub>A list of possible values for the field with a given name. A possible value contains two elements: `id` and `short_description`. The `id` is the value that will actually be submitted to an instance while the `short_description` is a more human readable string that describes what the value means. Specifying the list of `possible_values` is not required but if specified the list must contain at leas one element.</sub>\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 ServiceNow, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "SNulk provides a means to submit a large number of templated records to a table on a ServiceNow instance based on provided data. The table, instance, template, and data are all customizable and intended to be modified to fit the situation.",
"version": "1.0.1",
"project_urls": {
"Homepage": "https://github.com/ServiceNow/SNulk",
"Repository": "https://github.com/ServiceNow/SNulk"
},
"split_keywords": [
"snulk",
" servicenow",
" pysnc"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "25988472b81aa94bc3d7080467be7827347fb4e045164b3ee4dacbe140132d0a",
"md5": "d233dcac8c74010c70eef7f6aed81629",
"sha256": "d02ee988c796413e9b02e0442572e9aee6d431cfa10436614bd1256586188b8e"
},
"downloads": -1,
"filename": "snulk-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d233dcac8c74010c70eef7f6aed81629",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 37105,
"upload_time": "2024-10-23T16:15:56",
"upload_time_iso_8601": "2024-10-23T16:15:56.130138Z",
"url": "https://files.pythonhosted.org/packages/25/98/8472b81aa94bc3d7080467be7827347fb4e045164b3ee4dacbe140132d0a/snulk-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7f89aaf9efdf12cd0b85d9188622c2735a5cb030b7c27ef801a1493217ddc997",
"md5": "a178ab00aac9dd52becc4ab983c383f1",
"sha256": "80c484142ee1b90f3bb160175f7c926aed887db95102b260bcbc5e658e682e47"
},
"downloads": -1,
"filename": "snulk-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "a178ab00aac9dd52becc4ab983c383f1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 4924810,
"upload_time": "2024-10-23T16:16:00",
"upload_time_iso_8601": "2024-10-23T16:16:00.271951Z",
"url": "https://files.pythonhosted.org/packages/7f/89/aaf9efdf12cd0b85d9188622c2735a5cb030b7c27ef801a1493217ddc997/snulk-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-23 16:16:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ServiceNow",
"github_project": "SNulk",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "snulk"
}
