Start Here
The PacketViper Help Portal contains many of the Dashboards, Context Groups, and PCAP discussed in this document. Before beginning, please create a Help Portal account by visiting https://help.packetviper.com.
For customer locations which are air-gapped with no access to the Internet it will be necessary to first register and update all the PacketViper OTR devices and apply licenses prior to implementation.
OTR - PacketViper OT REMOTE (OTR) Security Solution for Distributed Operation
BSU - Boundary Security Unit. Located at the Internet/Enterprise IT Boundary
CMU - Control and Management Unit. Located on Internal Boundary
RSU - Remote Security Unit. Located at Remote Boundary.
Context - Including country, business, segment, and flow for network communications
Bridge - Transparent Interconnects between network segments
Decoy - Deceptive element configured to respond and mimic services and processes.
Sensor - Configured to detect, alert, and log anomalies.
Sirens - Deceptive elements that mimic device communication across the network
Threat - Malicious activities that intend to take advantage of network vulnerabilities.
Anomaly - Network behavior deviates from what is normal, standard, or expected.
Untrusted Device - An unknown IP detected (new IP device) on the network.
Untrusted Connection - An unknown network port detected on the network.
PacketViper OTR is a part of the PacketViper OT360 Security Solution that provides critical security capabilities at ICS boundaries and remote sites. OTR looks deeply into the network communications to ensure SCADA and other critical industrial controls operate more efficiently and securely. Our OTR solution is a turn-key and a simple to manage security solution that offers granular visibility, detection, and network control without the risk of disrupting normal industrial control processes. OTR provides operators with easy-to-understand dashboards and controls that clarify network communications and anomalies for operators without any prior network or security knowledge.
The OTR Solution is a combination of security software and hardware elements that allow industrial organizations to;
Protect industrial control processes locally and at remote locations
Provide granular visibility, monitoring, and managing of network communications from and to control systems and remote locations in real-time
Provide interactable security tactics which include decoy(s), sensors, and sirens that detect unauthorized devices or network communication within an industrial organization.
Provide operators with easy-to-understand dashboards that display network communication in context to quickly understand threats and anomalies
Provide a centralized management system to configure and manage remote locations
Provide an Enterprise Management interface to configure, and manage remote locations with industrial organizations.
Provide supervisors and operators with real-time alerting and reporting
The OTR solution architecture begins with a Boundary Security Unit (BSU), Control and Management unit (CMU), and Remote Security Unit (RSU) placed at remote locations. These units are connected using a simple network cable(s) to the existing network infrastructure.
The BSU, CMU, and RSU can be configured as a bridged connection that transparently monitors network communications or a mirror that receives a copy of the network communications from a smart switch.
Once implemented, PacketViper OTR monitors network communications between the BSU, CMU, and RSU(s), and aligns them to the customer's established security and control policies. These policies can be configured with an array of actions (log the event, provide an alert, send a message i.e. SMS, and take action/filter the network traffic).
Should the customer choose, the RSU(s) can be configured to contain anomalies to the location where the anomaly is detected to prevent the spread or access to other areas of the critical network environment. If configured for prevention, once the RSU detects a threat it creates a filter rule that prevents the unauthorized device or activity from communicating to the remaining environment, while simultaneously sending an alert, and notifying the CMU of a policy change. The CMU then notifies the remaining RSU(s) and propagates the filtering rule.
OTR is an industrial-strength solution designed to detect and prevent while contextualizing the network communication into simple-to-understand dashboard(s), and data for distributed environments. Without any prior experience, Operators can quickly visualize how network segments, and devices are communicating between one another with network layer level control through the Command and Management Unit.
PacketViper OTR consists of a family of appliances that operate together and consists of industrial-grade appliances that include rugged small-factor devices for remotes, and rack-mounted servers placed at plants. Boundary Security Units (BSU), and Control and Management units (CMU) are designed to align within plants and control systems servers such as SCADA live. Remote Security Unit (RSU) appliances are designed to be placed within remote locations and are designed to operate in harsh environments, with optional NEMA 3R containers that extend operating ranges.
Boundary Security Unit (BSU)
Is located at the outer boundary and designed to proactively detect external threats from the internet or enterprise IT sources. The BSU utilizes security capabilities such as moving target defense, monitoring, deception-based detection, context-based filtering, and vendor behavioral monitoring and control.
Control and Management Unit (CMU)
Is located on the internal network, and its role is to manage and monitor plant network communication and is the primary control manager for Remote Security Units (RSU). The CMU provides a one-to-many configuration management logic that offers a single point of configuration and management. The CMU also offers a quick method for the expansion of RSUs with no additional configurations needed for any additional OTR add-ons.
Remote Security Unit (RSU)
Is located at remote locations and is designed to withstand harsh environments. The RSU monitors remote network communication to PLCs and other devices which may be located at remote locations. It includes customizable dashboards that are simple to understand. RSUs are equipped with sensors, deception-enable detection mechanisms to detect anomalies, and alerting capabilities.
PacketViper OTR includes its own notification, logging, and containment system which operators can configure to alert, notify, or contain threats or anomalies. Once an anomaly or untrusted device is detected by a Remote Security Unit, a proprietary automated process transmits an alert, creates a new local security rule, and synchronizes the new rule with the Control and Management Unit. The CMU in turn will synchronize the remaining RSUs (at wire speed, typically seconds) thereby preventing them from accessing the remaining critical environment, and thereby containing the threat to a single location.
In the example below, when the threat is detected
(Follow along on the map below)
Local RSU detects an untrusted connection/device (threat/anomaly) and creates a security rule to prevent the potential threat.
Local RSU devices notify the CMU of a security policy change, and transmit the details of the threat.
CMU will transmit the updated security policy to the remaining RSUs within its scope.
* Local RSU devices will send a notification to Incident Response or SMS via email and write a log event of its actions and activity.
* Mail relay will need to be configured in order to receive alerts. This is discussed further in the document.
A fully deployed PacketViper OTR solution consists of a hardware device at each remote site that protects exterior and interior boundaries. Each device is configured with a transparent bridge, administrative port, sensor port and connected at the boundaries of each local site. The Remote Security Unit (RSU) devices are registered to the Control and Management Unit (CMU) from where the RSU are managed and configured.
The diagram below depicts a small site that consists of a Plant and two distributed sites. In this example below:
The BSU,CMU, and RSU are positioned at key boundaries through the environments;
Boundary Security Unit (BSU) - Between Carrier and Router. Proactively protects exterior boundaries
Control and Management Unit (CMU) - Between Switch and Router. Syncs RSUs, Monitors, detects, and prevents interior anomalies.
Remote Security Unit (RSU) - Placed at the remote location boundary inside or outside of the router at the remote location.
The Boundary Security Unit (BSU), Control and Management Unit (CMU), and each Remote Security Unit (RSU) will require;
Required to be registered to the PacketViper Secure Cloud
All Licensed applied
Receive the latest updates.
Two IP Addresses (Admin and Decoy/Sensor)
Four Network Cables (Admin, Decoy, Transparent Bridge)
Power outlets
BSU
Place your BSU within your rack. You will need a monitor, keyboard, and mouse.
Connect your administration port (ETH0), to the inside switch.
Connect the ETH1 sensor network port to inside the switch.
On the BSU bridge pair (ETH2/ETH3) connect your internet/enterprise IT to ETH2, then connect ETH3 to your router WAN port.
CMU
Place your CMU within your rack. You will need a monitor, keyboard, and mouse.
Connect your administration port (ETH0), to the inside switch.
Connect the ETH1 sensor network port to inside the switch.
On the CMU bridge pair (ETH2/ETH3) connect your router LAN port to ETH2, then connect ETH3 to your inside switch.
*Note: It is recommended to install your CMU first, before connecting the RSUs.
RSU
At the remote location connect your administration port (ETH0), and sensor network port (ETH1)
Router connected Remote: On the RSU bridge pair (ETH2/ETH3) connect your router LAN port to one side of the bridge, and the other bridge port to your core switch.
Switch connected Remote: On the RSU bridge pair (ETH2/ETH3) connect the connection from the plant to one side of the bridge and the other bridge port to your core switch.
One of PacketViper OTR's key attributes is the ability to provide context to network communications at wire speed. Context is critical to making fast and accurate decisions regarding network traffic conditions. Context is utilized to construct real-time dashboards to identify normal, suspicious, or malicious activity in the environment. Many different forms of Context are provided within PacketViper OTR. Basic context such as hostname, and labeling of network interfaces is vital.
Once the basic labeling has been completed you will find the hostname in the upper right-hand corner, and the network interface names list within widgets. As well you will be able to generate advanced analytics requests based on these names. It is important that this step is completed before we begin.
Labeling Network Interfaces on each device;
Go to Setup -> Networking.
You will see a list of interfaces ETH0, ETH1, ETH2, and ETH3.
Click on the ETH0 label and a pop-up will appear to enter the name. Limit the naming convention to six characters or fewer (example: admin, decoy, FW, WAN, LAN).
Repeat these steps for each interface.
Recommended Names
ETH0 - Admin
ETH1 - Decoy
ETH2 - Site (Represents traffic from site to plant)
ETH3 - Plant (Represents traffic inbound from the plant to the site)
Go to Setup -> Networking
Edit System Settings
Update Network Settings, and change the hostname
Click Save and Apply
The CMU (Policy Master/CMU) license will need to be applied to the CMU, and once applied you can build the enterprise as a single or multi-master with the CMU.
Single Master CMU:
Multi-Master Control and Management Deployments
A multi-master CMU deployment will need more consideration when utilized in this manner. The Location Masters are built within the CMU and designated during registration. Key considerations for choosing a multi-location master should be based on the similarities of remote networks, site roles, device types, and regional or geographical areas. The goal is to cluster distributed areas where policy sets would apply globally to that location cluster.
In a Multi-Master configuration, changes to policies will occur on the location master for the cluster, and not the CMU. Within the software and when adding RSUs you will need to designate one of the RSUs as the Location Master for each cluster. Once designated, register the remaining RSU to the location master.
Whether configured as a single master or a multi-master, each Remote Security Unit RSU will autonomously sync its policies to the CMU/Location Master within its scope. In either configuration, alerts are sent from the RSU when an event occurs and logging is stored locally. Policy changes for Cluster RSUs are made at the Location Master and not the CMU.
The Location Master for that site will send any updated policies to each RSU within its scope. The entire process should take no more than a few seconds depending on the connection speeds to its subordinate devices.
RSU detects untrusted connections
RSU creates a Local Policy to Block untrusted devices or connections
RSU Location Master will sync the new security policy to the remaining RSUs
Note: In addition, the RSUs will send alerts and logs event
Choose the CMU and log into it. Go to Setup->System and add the CMU License (this will require Internet access to PacketViper Cloud).
Once applied you will see a new Enterprise icon appear on the left-hand side
Click the icon, Create locations for all your sites or simply list them under the enterprise.
Click Add Host
Enter the IP of the RSU
Enter the user name and password of the RSU
Verify the RSU is correct.
For now, do not select a primary as the synchronization. Later, you can go back to the Enterprise Manager to select the Location Master
Add your Locations first (Organized by your need (Plant, Process, Division, City, State, Region, Country, etc.)
Once your locations are added, add your RSU to each location, and make sure you select the RSU as primary synchronization. All other RSUs added to these locations will sync to the designated location master.
Access the CMU
Go to Setup -> Mail & Alerts
Enter a unique email address <RSUname>@yourdomain.com
Enter Mail Relay “ mail.relayalerts.com”
Check Authentication
Enter Username and Password
Scroll to Enterprise Alerts and place a checkmark in the box
Enter a well-monitored email for alerts
Change notification time to 1-minute
Click Apply at the top.
Note: You will need to repeat the Enterprise Notifications steps for each RSU.
A PacketViper OTR trust is a known, verified, and expected communication between devices and networks. For example:
A SCADA system is a common process automation system that is used to gather data from sensors and instruments located at remote sites to transmit and display this data at a central site for monitoring purposes. The collected data is usually viewed on one or more SCADA Remote Terminal Unit devices located at the central or master site.
A real-world SCADA system can monitor and control hundreds to hundreds of thousands of I/O points. A typical Water Treatment SCADA application would monitor water levels at various water sources such as reservoirs and tanks. Consequently, when the water level exceeds a preset threshold, it activates the system of pumps to move water to tanks with low tank levels.
Common analog signals that SCADA systems monitor and control are levels, temperatures, pressures, flow rate, and motor speed. Typical digital signals to monitor and control are level switches, pressure switches, generator status, relays & motors.
While SCADA performs many tasks, the network connections within industrial operations are very similar for many control systems and much simpler to identify and trust within the PacketViper OTR Solution. In a typical IT environment, building trust-level policies is a much more challenging task that tends to become very complex as it grows due to the hundreds of different applications and services that must be considered. A trust in OTR terms is an exclusion or baselining of traffic and connections from the inspection process of a sensor, or eliminating the logging from a known, verified, or expected connection from being logged within dashboards.
Sensor Trusts (Exclusions) are used within Traffic Control->Sensors and by enabling the Invert option and thereby excluding trusted devices during the inspection process.
Custom Rule (Exclusions) are used within Traffic Control->Sensors and by enabling the Invert option and thereby excluding the trusted devices during the custom rule inspection process. These types of trusts could be used to exclude trusted activity to a decoy such as a network scanner.
Custom Rule (Do Not Log) can be used exclusively to not log known acts such as broadcasts or other communication such as DNS, NTP, and SMTP. Enabling Don Not Log provides less logging to the widgets.
Summary: Exclusions are generally reserved for control management devices communicating with subordinate devices such as PLCs. No Logging or Do Not Log are usually reserved for broadcast/multicast messages or for protocols like DNS and NTP.
Once an RSU device is connected to the CMU, you will need to build a dashboard to view network activity. This process will use widgets and filter them to specific protocols that will provide an easier way to visualize and understand the traffic flows
While remote site communication is normally low volume from Control Management servers to remote PLC devices, broadcast and multicast traffic typically represents the majority of the traffic observed on the dashboard and should be the focus when building trusts. Broadcast/Unicast/Multicast/Anycast traffic differs from network to network, however, many networks share common practices.
Unicast delivers a message to a single specific node using a one-to-one association between a sender and destination: each destination address uniquely identifies a single receiver endpoint.
Broadcast delivers a message to all nodes in the network using a one-to-all association; a single datagram (or packet) from one sender is routed to all of the possibly multiple endpoints associated with the broadcast address. The network automatically replicates datagrams as needed to reach all the recipients within the scope of the broadcast, which is generally an entire network subnet.
Multicast delivers a message to a group of nodes that have expressed interest in receiving the message using a one-to-many-of-many or many-to-many-of-many association; datagrams are routed simultaneously in a single transmission to many recipients. Multicast differs from broadcast in that the destination address designates a subset, not necessarily all, of the accessible nodes.
Anycast delivers a message to any one out of a group of nodes, typically the one nearest to the source using a one-to-one-of-many association where datagrams are routed to any single member of a group of potential receivers that are all identified by the same destination address. The routing algorithm selects the single receiver from the group based on which is the nearest according to some distance or cost measure.
Below is a list of some common broadcast/multicast IP addresses;
239.255.255.250
255.255.255.255
224.0.0.1
224.0.0.2
224.0.0.251
PacketViper OTR Trusts should always be created on the CMU unless configured as a multi-master. The CMU will sync policy changes to the RSUs.
There are several important steps that will need to be performed before creating the first dashboard that will simplify the trust creation process. The first step is preparing the CMU by creating Context Groups, and Custom Rule Groupings. These will be used to group devices and organize trust policies. It is important to note when building trusts to never build general or generic policies as one may find in other security appliances, such as firewalls. PacketViper OTR Trusts should be very specific and built to address a specific purpose.
There are different types of trusts we will configure within the CMU which will serve different purposes. Some Trusts will remove event logging and others will exclude connections from being evaluated by sensors while continuing to log the connection.
Sensor Exclusion Trusts: Used when the objective is to view traffic connections. Exclusions are used within Sensors and usually grouped within Context Groups. When creating exclusions within Sensors, the sensor will not evaluate the connection that exactly matches, and thereby permits the connection without incident to the next inspection phase (Custom Rules, Global Network Lists, Country).
Custom Rule Trusts: Normally used when defining broadcast/multicast connections. Because Broadcast/Multicast connection is highly trusted and voluminous, it is advisable to remove these log events from the dashboard - smaller haystacks make for more visible needles.
With the background provided above, it is time to implement the desired security strategy on the PacketViper OTR devices by accessing the CMU and RSUs and creating dashboards. This first dashboard is an important step in implementing OTR Trusts and requires several different widgets filtered to specific traffic types.
Note: You can download this and all dashboards from the PacketViper Help Portal here. Be sure to edit the widget filters to match your network(s) by clicking the gear on each widget.
Log into your remote device Click the +Add Dashboard and select Custom
Name the new Dashboard tab (Trust Build) by right-clicking on the tab, and entering the new name.
Click the Red + at the bottom right and add three separate Current Traffic Widgets.
You can move the widgets around by clicking and holding the four arrow icon. Organize them to provide visibility into the use case you are addressing. Below is an example Dashboard layout.
Now that we have properly filtered the widgets to specific protocols. It's time to build trust policies. The dashboards noted above are examples, and there is no limit on the number of widgets that can be created and filtered to provide visibility. It is important to note that in order to provide remote location customization capability, the CMU does not replicate dashboards. Therefore, any common dashboards created will need to be exported and then imported to each RSU if needed.
Right-click on Dashboard tab
Choose Export
Go to the other RSU, Add a new dashboard and choose Import.
Within the OTR solution there are customizable Context Groups that you will build for your environment. These context groups can be organized in any fashion. Context groups can be used to simplify trust and sensor creation.
While some trust relationships will be unique and require their own specific rules, most trust relationships will contain a Context Group to house groups of devices into a single group. Context Group provides a single point to add and remove devices as the network grows or contracts. Context Groups also provide a single rule to many devices and simplify administrative tasks.
In the steps below we will start with the IP addresses from our standard map and replace those IP addresses with your own. These Context groups should only be created on the CMU, and in turn, the CMU will replicate these Context Groups to the RSUs. An OTR Enterprise should have at a minimum the following Context Group;
Control Management Group: This network grouping will contain any management devices which control and monitor remote devices
Remote Devices: This networking group will contain an IP list of remote devices such as PLCs.
Remote RSUs: This networking group will contain all your RSU IP addresses
CMUs and RSUs: This networking group will contain all CMUs and RSUs
Broadcast IPs: This will contain a list of broadcast addresses you identify during your discovery
Inbound Network Port Group: This will contain a list of network ports destined for remote devices
Outbound Network Port Group: This will contain a list of network ports destined for remote devices
Sensor Excluded IP Addresses: This group will be used for sensors to exclude evaluations
Trusted Networks: will contain you local networks
Decoy IP Addresses: Contains all decoy IP addresses assigned to ETH1
Note: You can download and import all the Context Groups from our Help Portal here by selecting All Context Group.xml file. Be sure to edit each context group (video) to your network. You can also watch the video on how to create your own context groups.
Go to CMU and Log into unit
Go to Traffic Control->Context Groups. Create a new Context Group
Name it to your desire (example: Control Management)
Select IP / Network, and enter the “IP addresses” of the Management Servers
Click Save and your new Context Group will be seen in the list
Go to CMU and Log into unit
Go to Traffic Control->Context Groups. Click Create a new Context Group
Name it to your desire (example: Remote Devices)
Select the IP / Network button, and enter the “IP addresses” of the Remote Devices (i.e PLC)
Click Save and your new Context Group will be seen in the list
Go to CMU and Log into unit
Go to Traffic Control->Context Groups. Click Create a new Context Group
Name it to your desire (example: Remote RSUs)
Select the IP / Network button, and enter the “IP addresses” of the Remote RSUs
Click Save and your new Context Group will be seen in the list
Go to CMU and Log into unit
Go to Traffic Control->Context Groups. Click Create a new Context Group
Name it to your desire (example: Remote RSUs)
Select the IP / Network button, and enter the “IP addresses” of the CMU and RSUs
Click Save and your new Context Group will be seen in the list
Go to CMU and Log into unit
Go to Traffic Control->Context Groups. Create a new Context Group
Name it to your desire (example: Trusted Broadcast Group)
Select IP / Network Button
You can leave this blank for now. Click Save
Go to CMU and Log into unit
Go to Traffic Control->Context Groups. Create a new Context Group
Name it to your desire (example: Trusted Ports Inbound)
Select Ports Button
Enter ports in this format (example 23/TCP, 53/UDP). You can leave this blank for now. Click Save.
Go to CMU and Log into unit
Go to Traffic Control->Context Groups. Create a new Context Group
Name it to your desire (example: Trusted Ports Outbound)
Select Ports Button
Enter ports in this format (example 23/TCP, 53/UDP). You can leave this blank for now. Click Save.
Custom Rule groups are used to organize trust policies within the CMU. It provides a simpler method to manage policies across the OTR Enterprise.
Note: Watch the video on how to create your Custom Rule Groups
Log into your CMU
Navigate to Traffic Control->Custom Rules
Create the following groups by clicking the New Group button
Global
RSUs
CMU
Control Management
<Group for Each Remote Site>
Broadcast
Decoy Response
Sensor Blocks
In this document we will use the IP address found in the map example below. You can overlay this process onto your own environment to customize your trust. This process will require you to connect and open your CMU, and RSU in a separate browser window. You will observe and verify the traffic on the RSUs, and create the trust policy on the CMU as needed.
On the RSU we are noticing broadcast/multicast connections from the Control Management Servers Camera 10.1.1.12 and SCADA master 10.1.1.13. Our desire is to not log these connections and therefore we’ll need to create a Custom Rule trust to allow and disable logging for these connections.
Log Example
Source 10.1.1.13:52534/UDP Destination 239.255.255.250:1900/UDP
Source 10.1.1.12:52534/UDP Destination 255.255.255.255:138/UDP
Our first step is to add the destination IP address to the Broadcast Context Group and add the destination Ports to the Outbound Network Port Context Group.
TIP: On the CMU Open the Dashboard, Sensors, and Custom Rules areas in separate browser windows. This makes navigation much simpler during this phase.
Watch the video on editing Broadcast IP Group here
Navigate to Traffic Control -> Context Groups
Go to Traffic Control-> Context Group
Edit the Trusted Broadcast IP Group
Enter the broadcast destination IP Addresses and separate lines
Click Save and the Green Apply.
Our goal is to create our first policy that trusts broadcasts and does not log those messages on the dashboard from the control management servers. This step will stop the connection logging, and therefore not be evaluated by the sensors.
Watch the video creating the first broadcast trust policy here
Go to Traffic Control->Custom Rules
Click on the Broadcast Group on the left-hand side of the screen.
Click Add New Rule button
Click the Allow Radio Button
Enter the following Fields
Src/IP Network: Down Arrow and select Control Management Group
Dst/IP Network: Down Arrow and select Broadcast Group
Protocol: Select UDP
Src Port: Any
Dst Port: 1900
Enter a Descriptive Comment
Place a checkmark next to Do Not Log
Click Save, and green Apply
Once the Green Apply is clicked the CMU will initiate a sync to the remaining RSU. Normally this takes no more than 30 secs. However, depending on the connection speeds at the remote locations these times may be extended. You can monitor the sync through the CMU View Sync Queue.
TIP: To update any of the RSUs go to Traffic Control -> Custom Rules and click refresh. After a short period of time, the new policies will appear.
If you notice more broadcasts, simply add the destination IP to your Broadcast IP Context Group as you did earlier until there are no longer any broadcast connections observed in the dashboards. You can continue to the next steps and circle back as needed if more broadcasts are noticed. As a reminder, we are establishing these types of trust policies to remove expected traffic and events from the dashboards so that when an unexpected event does occur it is immediately noticed.
On the RSU device we can see the camera server is connecting to the remote camera at the remote site. The Camera Server (Control Management) communicates using port 39527/TCP to our cameras (Remote Device) 8080/TCP. Our desire is to exclude these connections so we can log these events on the dashboards. We will create a sensor that will include the context groups we created earlier in our sensor.
Example
10.1.1.12:39527/TCP 10.1.1.17:8080/TCP
10.1.1.12:39527/TCP 10.1.1.15:8080/TCP
Watch the video on editing the Inbound and Outbound Port Groups here
On the CMU
Go to Traffic Control-> Context Group
Edit the Inbound Trusted Ports Group
Enter the port of 8080/TCP,39527/TCP, and Click Save, and Green Apply.
Note: re-verify that the Source and Destination IP address are in the appropriate Context Group (example: Control Management, Remote Devices, Remote RSUs)
Watch the video on how to create a sensor with exclusions here
Go to Traffic Control->Sensors
Click Add New Sensors
Name the Sensor Remote Site Anomaly Detected Inbound
Enter the following Fields
Src/IP Network: Down Arrow Select Control Management and Select Invert
Dst/IP Network: Down Arrow Select Remote Devices and Select Invert
Protocol:
Src Port: Down Arrow Select Outbound Port Group and Select Invert
Dst Port: Down Arrow Select Inbound Port Group and Select Invert
Click Save, and green Apply
Once the Green Apply is clicked the Enterprise sync will be initiated and the update will be sent to the RSU device, These connections will still be logged, however, the sensor will not evaluate them. This will also apply to all sites simultaneously.
.
Example
10.1.1.13:5555/TCP 10.1.1.16:4061/TCP
10.1.1.13:5555/TCP 10.1.1.14:4061/TCP
On the CMU
Go to Traffic Control-> Context Group
Edit the Outbound and Inbound Port Group
Enter the ports of 4061/TCP, and 5555/TCP in each and Click Save, and Green Apply.
Click Save, and green Apply
Once the Green Apply is clicked the Enterprise sync will be initiated and the update will be sent to the RSU device. These connections will still be logged, however, the sensor will not evaluate them. This will also apply to Site 2 simultaneously.
We also noticed the PLCs (Remote Device) communicate outbound to the Control Management servers. We will need to add these ports to the Outbound Port Group and Create a new sensor to detect untrusted outbound connections.
Example
10.1.1.16:4061/TCP 10.1.1.16:5555/TCP
10.1.1.14:4061/TCP 10.1.1.14:5555/TCP
On the CMU
Go to Traffic Control-> Context Group
Edit the Inbound and Outbound Port Group
Enter the port of 4061/TCP, and 5555/TCP in each.
Click Save, and Green Apply.
Go to Traffic Control->Sensors
Click Add New Sensors
Name the Sensor Remote Site Anomaly Outbound
Enter the following Fields
Src/IP Network: Down Arrow Select Remote Device Group and Select Invert
Dst/IP Network: Down Arrow Select Control Management Group and Select Invert
Protocol: Any
Src Port: Down Arrow Select Trusted Outbound Port Group and Select Invert
Dst Port: Down Arrow Select Trusted Outbound Port Group and Select Invert
Click Save, and green Apply
Once the Green Apply is clicked the Enterprise sync will be initiated and the update will be sent to the RSU device. These connections will still be logged, however, the sensor will not evaluate them. This will also apply to Site 2 simultaneously.
Now that we have created the Sensor, Custom Rules, and Context Group trusts, we will need to create a dashboard on the Remote Site RSU device which monitors the Sensor activity. This dashboard can also be used to ensure the sensors have been built correctly. Note: Dashboards are not replicated from CMU, however, they can be exported from one OTR device to another.
Note: You can download this dashboard from our help portal here and select the Decoy_Sensors.json file. Be to edit the widgets filters to your network settings by clicking the gear on each widget.
On your RSU device go to the Home Screen and create a New Dashboard
Name your Dashboard Sensor
Click the Red + at the bottom right-hand corner
Select Current Traffic widget
Click the gear icon on widget
Go to Filtering Reason
Select Sensor option
Choose one of the two sensors you created
Click outside the filter to close
Rename the widget to match the sensor you select
Repeat the same process for the second sensor
If the trusts have been created properly there should be no activity in the sensor widgets. If you are noticing sensor alerts, we may have missed a few connections that should have been added to trust relationships. Use the same techniques as we used in Building Trust Policies to add additional exclusions to the Sensors, or create additional Custom Rule if the connections are valid and not required to be logged.
Once you have trusted all the connections we are ready to test the sensor. From a browser-capable workstation within the critical environment.
Open a browser
Have the RSU dashboard open
Enter the device's IP address and append it with a unique port number.
Example http://10.1.1.13:3333
On the RSU dashboard that is monitoring the device, you are testing you should see the alerts appear in the widget
[Your Source IP] to 10.1.1.3:3333/TCP
Repeat this process for each device and sensor.
A decoy is an essential tool for the OTR security strategy. They provide the ability to detect laterally moving traffic within the segment and lean into the threat by mimicking and responding to actual network services such as www, mail, FTP, and many other things. Decoys are paired with Sensors to Alert, Prevent, and Block any untrusted device or connection. For this document, we will create a simple web page that will respond to a browser request for Wago CODESYS – TCP: 2455. We will use the 2455/TCP network decoy port.
You should have already connected a network cable(s) from the decoy port on the Remote OTR to the remote switch, and assigned it an IP address for that segment. To assign an IP address to a Remote RSU;
Log into the Remote RSU
Go to Setup->Networking
Scroll to ETH1 interface
Single-click the ETH1 label and rename it to Decoy and click OK
Click the Update IP settings button
Statically assign the network information.
Uncheck the Use System Gateway option
Click Save and Green Apply
Before we begin we will need to create a Context Group which lists all the assigned decoy IP addresses. This Context Group needs to be created on the CMU so it may be replicated to all Remote RSUs.
Note: You can download this context group from our help portal here and download the Decoy IP Addresses.xml file. Be sure to enter the networks and IP addresses listed.
Log into your CMU or Location Master
Go to Traffic Control->Context Groups
Create a new Context Group, Enter the name of Decoy IP addresses, and select IP/Network
In the list box that appears, enter all the Decoy IP addresses of all Remote RSUs. Enter the IP addresses one per line.
After clicking Save, click the Green Apply button
It’s now time to create the decoy response within Custom Rules. We will use the 2455/TCP as the responding network port. In addition, we will add a simple web response and if a connection request is made to [Decoy IP]:2455/TCP the response will be displayed, or establish an SYN/ACK. The decoy(s) will also respond to other network requests such as probes.
Go to the CMU or Location Master
Go to Traffic Control->Custom Rules
Click Add Rule
Click the Allow Radio button
Change the Priority to 0
Edit the Destination IP/Network by clicking the down arrow and selecting Decoy IP Addresses Context Group
Select the Protocol of TCP
In the Destination Port enter 2455
Enter a brief description in the comments field Example Remote Decoy Response
Scroll to the bottom of the Rule and click the Select button next to the Decoy Response
Click New HTTP response
Enter and Response Name
In the Body, enter “Redirecting Wago CODESYS….”
Click Save and your new response will appear in list
Click the check box next to it, and you be returned to Rule
Click Create, and Click the Green Apply Button
Using a browser-capable workstation within the same network segment. Simply open a browser window and enter the address HTTP://<decoyip>:2455. Once connected you will see Redirecting Wago CODESYS…. on the browser.
Once we have created the Decoy Response Rule it is time to build the corresponding sensor. The sensor is responsible for the actions imposed on the untrusted connection (log, alert, slow down, prevent).
Similar to the other Sensors we created earlier, we will create this sensor with the destination IP Decoy IP Context Groups and TCP port 2455.
Log into CMU or Location Master and go to Traffic Control->Sensors.
Click the New Sensor button and create the following sensor in this manner.
Sensor Name: <Site Identifier> Decoy Activity
Sensor Position: 1 - Before Custom Rules
Log Event: Checked
Bi-Directional: Unchecked
Source IP/Network: Any
Destination IP/Network: <Decoy IP>
Protocol: TCP
Source Port: <Leave Blank>
Destination Port: 2455
Rate Triggered: Unchecked
Source Geo-IP/Context Group: <Leave Blank>
Destination Geo-IP/Context Group: <Leave Blank>
Schedule: Always Active
Now that we have finished our decoy and sensor for our remote RSUs, we will need to create a separate dashboard to monitor any activity. For this dashboard, we will require three Current Traffic widgets;
Monitor all activity to the Decoy IP
Monitor the Sensor Activity
Monitor Decoy Responses
Note: You can download this dashboard from our help portal here by downloading the Decoy_Sensors.json file. Be sure to edit the widget filters to your settings by selecting the gear at the top of each widget.
Each Current Traffic widget will be filtered to a specific Decoy IP address and sensor. Because dashboards do not replicate through the Enterprise Sync, we will need to Export, then Import the dashboard to each Remote RSU
Log into your Remote OTR and go to Dashboards;
Click + Add Dashboard
Decoy Activity Widget
Click: Red +
Select Current Traffic Widget
Rename the Widget to Decoy Activity
Click the icon of that widget
In the SRC AND Dst IP field, enter the Decoy IP address
Click outside the filter to close the filter
Sensor Activity Widget
Click: Red +
Select Current Traffic Widget
Rename the Widget to Decoy Sensor Activity
Click the icon of that widget
In the Filter Event area, Select Sensor
Select the Decoy Sensor Activity sensor
Click outside filter to close the filter
Decoy Responses Widget
Click: Red +
Select Current Traffic Widget
Rename the Widget to Decoy Sensor Activity
Click the icon of that widget
In the Filter Event area, Click Any and Select Decoy Response
Click outside filter to close the filter
From the network the decoy is connected to
Connect a separate laptop or workstation.
Open the browser and enter the following in the address bar
HTTP://<decoy ip>:2455
Go to the dashboard of the RSU decoy you are testing and navigate to the decoy/sensor tab.
In the tab you will see the activity of the connection request in the sensor widget.
This will signify the sensor has detected the connection to the decoy
All of the assets in the PacketViper OTR deployment are capable of producing and supporting Sirens as part of the deception strategy. The Siren is a deceptive tactic that produces and transmits false traffic from one point to another on the network. By simply uploading a PCAP file of a captured session you are able to create false - but highly believable - traffic on the network. The primary purpose and use case for sirens is to address those attackers and deeply embedded threats which may be lying in wait on the network and monitoring traffic, in order to lure them out of hiding to the decoys and sensors.
When building Sirens it's important to note that the sources and destinations are valid. For this reason, we will use the IP addresses of the decoy addresses we used earlier. The falsehood in the traffic comes within the packet itself. The packet details may represent anything you choose to send.
When capturing session traffic on the wire, generally there is a sending request, and shortly thereafter a response for the request from the destination.
We are using this same methodology in order to provide the most realistic traffic sets. We configure the Sirens in the same form, and for this reason, we build Sirens in pairs, Source-Sender and Destination-Responder.
Source-Sender: Emulates the sending portion of the session and is usually performed by the CMU
Destination-Responder: Sends the response portion of the session and is usually performed by the RSU
Note: On our Help Portal we have provided several PCAP files which can be used for Sirens. Each file is appended with “sender” or “responder”. You can visit https://help.packetviper.com and search for PCAP.
The Control and Management Unit (CMU) provides the sending packets and strategically designated Remote Security Units (RSU) will provide the responding packets. Using this technique provides the most realistic appearance for the deceptive traffic.
When configuring a Siren it is important to note the destination IP address to where the session PCAP needs to send the request. Then at this destination you place the responding PCAP and send it back to the CMU
Go to help.packetviper.com and download the iec104-Sender.pcap and the iec104-Responder.pcap files from https://help.packetviper.com/portal/en/kb/articles/siren-pcap-and-tools
Once downloaded, log into your Control and Management Unit (CMU)
Go to Traffic Control-Sirens
Click on Add Siren, Enter the name IEC104 Sender
Enter the Source IP of the CMU Decoy
Enter the Destination Ip of the RSU decoy IP
Select the Decoy IP interface
Click on the Packet Source and upload the iec104-Sender.pcap file
Make any necessary changes and click apply
Leave the Frequency at 10
Enter any comment and click Apply
Note: Once applied you will notice the Siren is running. You can pause and edit the Siren as needed.
Log into the RSU for which you designated the destination on the sender
Go to Traffic Control-Sirens
Click on Add Siren, and Enter the name IEC104 Responder
Enter the Source IP of the RSU Decoy
Enter the Destination IP of the CMU decoy IP
Select the Decoy IP interface
Click on the Packet Source and upload the iec104-Responder.pcap file
Make any necessary changes and click apply
Leave the Frequency at 10
Enter any comment and click Apply
Note: Once applied you will notice the Siren is running. You can pause and edit the Siren as needed.
Important to Note: Be sure to exclude these types of sessions on your sensor context group or you will see receive alerts from the Siren sessions.
Once you have created your new Dashboards, you can export them to a local drive, and import them to other Remote OTR devices. There are a few caveats to note when exporting and importing;
Widgets that have specific IP addresses configured within the Src AND Dst IP/network, such as the Decoy Activity Current Traffic Widget which is configured with the Decoy IP of the specific RSU in this document. These types of widgets will need to have the IP address changed for each RSUonce imported. The Src AND Dst IP/Network will need to reflect the correct IP for that Remote RSU
Log into the RSUyou wish to export the dashboard from
Right-click on the Dashboard Tab to Export
Select Export
Save the file
Log into the RSU to import the new dashboard
Click + Add New
Select Import
Choose your file
Change any static IP address within the widget (Example: Decoy Activity widget)
Every sensor can be configured to send an email alert when an untrusted device or connection is discovered. These alerts can be customized to send specific details to the Incident Response teams or any appropriate contact person in the organization. There are a few prerequisites for sending email from a critical distributed environment;
Direct Send - The remote RSUs are capable of establishing an SMTP session to send mail. This will require DNS to resolve domain names.
Mail Relay or Local Mail Server- If DNS is not available, the RSU will be required to relay its messages through a relay host. This host could reside in the critical distributed environment, or the RSU will be required to access a mail relay server outside the environment.
In any of the above scenarios, system mail settings will need to be configured. It is important to have the sending address unique to the Remote RSU.
To configure the RSU you will need to log in to each RSU and configure the mail settings for that unit. These settings are not replicated through the CMU and must be configured on each device.
Direct Send
Log in to the Remote RSU
Go to Setup -> Mail and Alert Setup
Enter a unique email for the Remote RSU
Click Apply
Click Green Apply
Click Send Test Message
Enter email and send
Mail Relay Using PacketViper Relay Server
Log in to the Remote RSU
Go to Setup -> Mail and Alert Setup
Enter a unique email for the Remote RSU
Enter and mail relay addresses of mail.relayalerts.com
Click Authenticate to Relay Host
Enter the username and password provided to you
Click Apply
Click Send Test Message
If your message was received then your RSU is ready to send alert messages. If no message was received, check location firewalls for port access, and make sure your mail server has whitelisted the sending address.
Now that we have configured our sensors, and ensured all trusted devices have been accounted for with trust relationships, we will need to configure the email alerting mechanism for the sensors. This is an important aspect of the solution to provide the notification of any action taken by the OTR environment. While notification is strongly recommended, it is important to remember that the ORT solution will still perform vital functions such as (prevent/replicate) should an untrusted device or connection be detected in the absence of notification.
If email is not supported outside the critical environment, we recommend a dedicated console or workstation which can continuously monitor the dashboards of the OTR solution. The dashboards can display alert notifications.
If your environment supports email alerting outside the environment, we recommend configuring an email alerting action within the sensors. You can add multiple email notifications to the sensors by simply adding additional actions to the sensors.
Log in to the CMU
Go to Traffic Control - Sensors
Click the edit icon next to the sensor to edit
Click the + Action
Select Send Email Alert
Enter the email address
Send Every: 30
Click select Email Button
Click New Email Button
Enter a Descriptive email subject
Edit the email body. By default, notifications include Time, Src Country, Src IP, and Dst Port which are displayed within the body of the alert. You can select the entire contents and delete them.
Once deleted, click inside the email body field
Below is the email body field, you can select options to create a customized template. Choose any applicable option
Example
Sensor Name: %%SENSORNAME%%
Time: %%SENSORTIME%%
Src IP: %%SRCIP%%
Dst IP: %%DSTIP%%
Once you have created your email message, Click Save
Select the next to the template to use for sensor
Save the Sensor, and click the Green Apply.
PacketViper provides the ability to selectively send sensor information to our secure cloud called Alertbox. Alertbox is different and incremental to the local reporting system with OTR and its primary focus is to correlate sensor information into an active report. Many clients leverage Alertbox reporting for executive and summary reporting requirements. Normal reporting provides details on all aspects of the activity such as Allows, Blocks, Sensors, Responses, and other minute information
Alertbox focuses on the sensor activity to provide a crisper and less convoluted report to our customers such as;
Geo Hot Spot
Attacker Focus
Vendor and Business
ASN History and trends
Threat details
Trends, and Risk
Inbound, Outbound
Alertbox requires a License key, and in order to use it requires access to the PacketViper secure cloud through your firewall.
To configure Alertbox you must first obtain a license key and apply it to all units;
Go to Setup->System
Version
Paste License Key
Once the license has been applied, you are ready to add the action to your sensors
Go to Traffic Control -> Sensors
Edit the Sensor to send data to Alertbox
Click the Add Action button
Select Alertbox
Enter your custom details
Click Save
Click Green Apply
Once we have completed the email alerting and assigned the correct email templates to our sensors. We can now add the actions to automatically prevent untrusted connections or devices from accessing other areas of the critical environment by creating a policy automatically on the RSU.
To organize automated policies we will create Custom Rule groups within the Traffic Control->Custom Rules area as a destination for the newly generated custom rules.
Log in to your CMU
Go to Traffic Control -> Custom Rules
Click New Rule Group
Name the Group to align it with your sensor name
Repeat this process for each sensor that you will be adding auto-blocking action
Log in to your CMU
Go to Traffic Control -> Sensors
Click the edit icon next to the sensor to edit
Click the Add Action
Select Add Src to Custom Rules Blocking (Sensor Block)
Add a short descriptive comment
Select the Custom Rule Group that matches the sensor
Select a Time frame (recommended 7 days)
Click Save and Green Apply
Repeat the steps for each Sensor to auto-prevent threats.
It is now time to test the OTR solution. The test will demonstrate the OTR solution's ability to detect and prevent untrusted connections for any location the OTR solution is deployed. We can perform these tests using several different methods;
Plant: From the plant, we will attempt to connect to a remote location using an untrusted connection, or device.
Remote Location: Connect to the remote location network switch, and attempt to connect to other areas of the network.
Equipment Required
Laptop
IP Addressable NIC Port
Network cable
Browser, Putty, Telnet, Command-Line
We will configure the testing laptop with either a unique IP address or the IP address can be obtained from DHCP. We will use this laptop to connect to the switches at the plant and a remote site. From each of those locations, we will connect our testing laptop and make attempts to connect from the plant to the remote sites, and from the remote sites to the plant.
In the document, we will begin our test from the plant. We need to monitor the RSU dashboards from a trusted workstation (other than the test laptop). Your dashboard should open to the Sensor dashboard which monitors untrusted activity.
We are now ready to connect the test laptop to the Plant switch. Once connected, you can use any tools we mentioned above to perform your test. In this test, we will attempt to connect to the PLC via a browser and TCP/80.
From the test laptop launch the browser application and type in the PLC address http://10.1.1.14 and press enter.
If you have configured the email alerts check your inbox for the alert from the Remote RSU
On the RSUdashboard you will see the events
If you have configured autoblocking, go to Traffic Control->Custom Rules->Sensor Custom Rule Grouping. You will see a new policy created with the IP address of the Test Laptop added to the block.
On the CMU, go to Traffic Control->Custom Rules->Sensor Custom Rule Grouping, You will see the same policy with the Test Laptop IP address.
Go to a different RSU. Go to Traffic Control->Custom Rules->Sensor Custom Rule Grouping. You will see the same policy with the Test Laptop IP Address.
The RSU detected an unauthorized connection attempt to the PLC. The RSU created a new policy that was replicated to CMU and promulgated to the remaining RSUs. The laptop is now prevented from leaving the Plant through the firewall to outside sources and is no longer permitted to connect to any PLCs protected by an RSU.
Before we can begin the next test we will need to reset the environment and remove the blocking rules created when the test laptop was detected in the first test.
Disconnect the test laptop
Go to the CMU
Go to Traffic Control - > Custom Rules
Go to the Sensor Custom Rules Grouping
Click the trash can next to Test Laptop Policy
Click the Green Apply
After a few seconds the CMU will sync the update to all RSUs and you are ready for the second test.
For this test we will need to be physically present at the remote location to connect the Test Laptop to the remote switch. For this test, nothing on the test laptop will need to be opened;
Connect the test laptop to the switch at any remote site which is protected by an RSU.
On the Sensor dashboard, you will notice activity in the new device widget.
Log in to your laptop
Go to the RSU home screen
On the Widget which monitors for untrusted connections, you will see a log event for the laptop you connected.
If your sensor was configured to alert, you should have received an email on the event.
If the sensor was configured to contain threats, Go to Traffic Control->Custom rules and you will see a new rule created with the IP of your laptop.
If you open a browser on your laptop and attempt to connect to PLC outside the remote site, you will be denied access.
Attempt to ping other devices outside the remote site, and the pings should fail.
EFFECTIVE OT/ICS SECURITY with PacketViper OT360 / PacketViper OTR
Securing Operational Technology (OT) and Industrial Control System (ICS) assets requires more than the visibility of unpatched assets or mere detection of anomalies. Effective cybersecurity depends on providing front-line operators of critical operations the required information to act when potential threats are in motion. PacketViper OTR is designed and configured for operators and provides industrial-level defense and protection in an easy-to-use format. Our unique approach ensures that sensitive and complex devices are not disrupted.
RAPIDLY SECURE, MONITOR, DETECT, and PREVENT THREATS IN A CRITICAL INDUSTRIAL OPERATION.
Operational technology (OT) and Information Technology (IT) systems have converged in order to optimize production, drive innovation and increase efficiency. However, that convergence increases the attack surface by connecting network segments that had previously been air-gapped and exposing them to broader networks and the Internet. Detecting complex and evolving cyber threats requires advanced tools, knowledge, and training. The PacketViper OTR Solution utilizes easy-to-understanding dashboards allowing operators to monitor status in real-time, without the need for advanced cyber security knowledge or experience.
ENTERPRISE-WIDE VISIBILITY, MANAGEMENT, AND CONTROL
Whether your organization is a small one-site location or a large multi-site complex and distributed environment, detecting and then stopping an attack is difficult without the correct tools. PacketViper OTR provides Industrial level enterprise visibility, security device management, threat mitigation, and containment in a single solution. The OTR solution eliminates blind spots, and network risk regardless if the environment is flat or segmented. Granular and easy-to-understand dashboards provide real-time views of devices, rapidly identifying threats, weaknesses, and risks from potentially vulnerable devices within critical industrial control systems. The PacketViper OTR solution is control system platform agnostic and requires no additional tools to detect and defend Industrial control systems.
SECURING OT, ICS, and SCADA ENVIRONMENTS
We understand security, and the many differences between IT and OT/ICS/SCADA networks. We also know that effective cybersecurity requires a layered approach in any environment. Our mission is to provide effective, easy-to-deploy, and affordable solutions that defend and maintain the availability of ICS and OT networks, including how they interface and connect with IT infrastructure. Finally, you have a security solution for your critical infrastructure that protects your OT Network and its many components, including HMIs, PLCs, RTUs, SCADA assets, Historians, and more.
About PacketViper
PacketViper provides transformative and trusted cybersecurity solutions for organizations seeking to modernize the cybersecurity of converging OT and IT networks and defend distributed OT endpoints. PacketViper’s OT360, OTR, and Deception360 products deliver an agentless detection, prevention, and response technology that automate attack prevention from both external and internal threats. PacketViper customers cover multiple public and private sector industries. For more visit packetviper.com.