Search Knowledgebase

Follow

Download XMLs From ICISDA Dataflow to Correct ICIS Data.

New parameter fields are available as of June 18th, 2020:

GetICISDMRViolationData (Refer to the following screenshot):

RNCDetectionCode
RNCDetectionDateBegin and RNCDetectionDateEnd (creates a date range for the RNCDetectionDate)
RNCDetectionFlag ('detected', 'true', 'yes', or 'y' will return ONLY records that contain ReportableNonComplianceDetectionCodes versus 'not detected', 'false', 'no', or 'n', which will return ONLY records that do NOT contain any ReportableNonComplianceDetectionCode.
RNCResolutionCode
RNCResolutionDateBegin and RNCResolutionDateEnd (creates a date range for the RNCResolutionDate) 

GetICISDischargeMonitoringReportData:

NODICode

Retrieve ICIS-NPDES data from the 45 ICISDA ( ICIS Data Access ) web services at EPA's Exchange Network Services Center (or through your own node software, OpenNode2, etc.). You could then use the data for your agency's system (performing quality assurance, seeding Exchange Network plugins with data, deriving anticipated DMRs for eDMR systems) or resubmit XMLs back into ICIS using the EDT batch process (ICIS-NPDES dataflow) after performing find/replace and other edits. You could also save retrieved XMLs for testing or training purposes and submit them to the ICIS Test or Stage environments after a data refresh to restore your data. 

 

Registration

To download from the ICISDA dataflow (ICIS Production data), you will need to register for a CDX account, if you don't have one already, through the Request an Account link on the aforementioned ENSC homepage and choose ICISDA for the question "What DataFlow will you primarily be submitting to?". 

 

Hours of Operation

ICIS Nightly Processing begins around 11PM EST when Prod ICIS data is replicated to ICISCOPY, among other processes. Therefore, please do not query or schedule jobs from 11PM to 7AM EST, when the jobs are usually completed.

 

Production Environment

For manual users of ICISDA, login to the Exchange Network Services Center at enservices.epa.gov. Within the Exchange Network Services tab, search for any of the services listed in the following Documentation section , or simply search 'ICISDA', which will list them all. The Production ICISDA services have a Dataflow value of ICISDA and a Node value of .NetNode2 (Refer to the following highlights). ICISDA Production retrieves data from the ICIS Copy database, which in turn is a nightly copy of data from the ICIS Production database.

 

NodeAndDataflow.jpg

"A screenshot of the 'News and Data Channels' tab within 'Exchange Network Services' highlighting the 'ICISDA' Dataflow and '.NetNode2' Node."

 

Test Environment

You may also register for the Exchange Network Test site: enservicestest.epacdxnode.net, where the Test ICISDA services connect to the ICIS Stage database (not ICIS Test). Whereas the data in ICIS Copy is refreshed nightly, ICIS Stage, is usually refreshed with Production ICIS data about every three months. Test ICISDA can be used to retrieve test or training records that you want to save before a data refresh/wipeout to ICIS Stage, which can then be submitted back to ICIS Stage using the inbound ICIS-NPDES dataflow on the .NGNTest2.0 Node (not .NGNDev2.0. Refer to the following screenshot) after the refresh. Saving these templates and modifying them for future use can save a lot of time instead of manually reentering them in the ICIS Stage website.

For guidelines on retrieving NetDMR training/test data from ICIS Stage (before a database refresh), visit: 
netdmr.zendesk.com - ICIS Data Access.

 SharedScreenshot.jpg

"A screenshot of the 'News and Data Channels' tab within 'Exchange Network Services' highlighting the 'ICISDA' Dataflow and '.NetNodeTest2.0' Node."

 

Documentation

The Flow Configuration Document has technical details of the ICISDA dataflow.  This can be found at exchangenetwork.net/data-exchange/icis-data-access.

The following is a list of the 38 services that return XML results that generally conform to the ICIS-NPDES XML Schema format and 7 services that return XMLs with common ICIS ref table codes and descriptions.

ICISDA services returning ICIS-NPDES data families ICISDA services returning ICIS reference tables
GetICISBasicPermitData
GetICISBiosolidPermitData (deprecated version as of 2018)
GetICISBiosolidProgramReportData
GetICISCAFOAnnualReportData
GetICISCAFOPermitData
GetICISComplianceMonitoringData
GetICISComplianceScheduleData
GetICISCSOEventReportData
GetICISCSOPermitData
GetICISDischargeMonitoringReportData
GetICISDMRViolationData
GetICISEnforcementActionMilestoneData
GetICISFederalComplianceMonitoringData
GetICISFormalEnforcementActionData
GetICISGeneralPermitData
GetICISInformalEnforcementActionData
GetICISLimitSetData
GetICISLocalLimitsReportData
GetICISMasterGeneralPermitData
GetICISNarrativeConditionScheduleData
GetICISParameterLimitsData
GetICISPermitTrackingEventData
GetICISPermittedFeatureData
GetICISPOTWPermitData
GetICISPretreatmentPerformanceSummaryData
GetICISPretreatmentPermitData
GetICISScheduleEventViolationData
GetICISSingleEventViolationData
GetICISSSOAnnualReportData
GetICISSSOEventReportData
GetICISSSOMonthlyEventReportData
GetICISStormWaterConstructionPermitData
GetICISStormWaterEventReportData
GetICISStormWaterIndustrialPermitData
GetICISStormWaterMS4LargePermitData
GetICISStormWaterProgramReportData
GetICISStormWaterMS4SmallPermitData
GetICISUnpermittedFacilityData
GetICISAffiliationModuleCrossReferences
GetICISParameters
GetICISPollutants
GetICISUnits 
GetICISUnitConversions
GetICISUnitGroups
GetICISUnitGroupCrossReferences

"A table of ICISDA services that return NPDES data families and ICIS reference table XMLs."

 

For more information about the inbound ICIS-NPDES dataflows (EDT Batch submissions), visit: Guide To icis.zendesk.com - Submitting Dataflows To ICIS .

Presentation from a 2016 ICIS-NPDES monthly call with some instructions for sample service requests.

 

Hints and Tips

For string parameters or fields (not dates, integers, etc.), the string can be partial matches, e.g., ‘TX’, ‘tx’ (case is insensitive), or ‘TX003’ for NPDESPermitNmber, searches for all occurrences of this substring within the parameter (‘%tx%’ will also search for ‘tx’ anywhere in the string, whereas ‘tx%’ will search for the occurrence of this term only in the beginning of the string).

A code-type parameter (usually ending with ‘Code’), as well as some NPDESPermitNumber parameter, can accept a string with multiple values separated by commas with or without spaces (because the string is passed to a SQL IN operator). Other text parameters, including ZipCode (although it has ‘Code’ in the name), do not qualify, because they are not part of reference code lists in the ICIS database.

The NPDESPermitNumber parameter can also accept a comma separated string of up to 2000 characters if the query is a solicit-type service (Refer to following screenshot) listed in Section 3.3 of the Flow Control Document. The NPDESPermitNumber field in query-type services does not accept comma-delimited values.

 DMR Violation parameters:

DMRViolations.jpg

"A screenshot of the News and Data Channels tab highlighting the NPDES Permit Number, RNC Detection Code, and Violation Code fields."

 

Older data returned from ICIS might not conform exactly to the current Schema, so if the intent of the user is to submit the XML back into ICIS, the user might need to edit the non-conforming data. For example, if a ScheduleViolationCode is returned in the GetICISComplianceScheduleData service, it might contain ‘C10’ or ‘C20’ values from legacy PCS, but the ICIS Schema only accepts either ‘C30’ or ‘C40’. Another example is that some <EnforcementActionGovernmentContact> blocks returned in the GetICISFormalEnforcementActionData service, might not contain any <ElectronicAddressText> data, which is required by the ICIS XML Schema.  

Similarly, ICISDA mostly returns the raw data values stored in the ICIS database, which might not conform to the ICIS Schema. For example, the <NumberOfReportUnits> values returned in GetICISBiosolidProgramReportData could be ‘5’, but the ICIS Schema wants ‘05’, so it is up to the user to convert the format to whatever system they are working with- their own system or ICIS or something else. The only exceptions to this raw data return are the values in the PermitStatusCode and StatusCode tags for Basic and General Permits as detailed in the FCD.

The <Header> tag at the beginning of all returned XMLs will have a default <Id> value of ‘ICISDA’. If you plan to edit and submit the XML back into ICIS, replace this with your EDT approved ICIS User ID. The <Property> block is included with the values 'Source' and 'FullBatch' in the <name> and <value> tags so that ICIS EDT returns a Response PDF with the Acceptance and Rejection details of your submission. For example:

<Header>
  <Id>ICISDA</Id> <!-- Replace with your EDT User ID -->
  <Property>
    <name>Source</name>
    <value>FullBatch</value>
  </Property>
</Header>

GetICISParameterLimitsData includes a parameter called IncludeAllVersions, which defaults to False if not entered.  Enter 'true' (case insensitive) if you want to return records from previous versions of the permit.

Four other services return data from ALL versions of a permit by default, including previous versions: GetICISDischargeMonitoringReportData, GetICISDMRViolationData, and GetICISScheduleEventViolationData, GetICISSingleEventViolationData. If you do not want data from previous permit versions, please enter the appropriate Begin and End Dates to create your date range for Monitoring Period End Date (MPED), Schedule Event Date, or Single Event Violation Date. The rest of the services return only the current version of permits.

For federal compliance monitoring data, use the GetICISFederalComplianceMonitoringData (State_EPA_Flag = 'E') instead of GetICISComplianceMonitoringData (State_EPA_Flag = 'S'), which returns state compliance monitoring data.

 

Article revised on 10/25/2021.

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request