Lifting an electric vehicle charger into the cloud

With the increasing popularity and availability of battery electric vehicles, privately-owned charger infrastructure at home or on company premises has become more and more common.

A vehicle charger is much more than a simple mains socket – it contains a computer or at least a programmable logic controller (PLC) which keeps track of power levels, consumption, and state. Having access to this data can be very useful for the owner, for example for custom billing or energy management purposes. Existing proprietary solutions offered by the manufacturers do not necessarily guarantee a full match with the customer’s requirements regarding costs and connectivity.

So, let’s “rescue” the data from the charger and store it in our own infrastructure.

This article serves two primary purposes. First, it outlines a simple solution on how to acquire charger data for your own analysis purposes. Second, it can be used as a tutorial for a tiny IoT use case with Golang, MQTT and TimescaleDB.

Obtaining data from the charger

At the time being, most chargers provide a management interface using a proprietary protocol with varying degrees of documentation. In case of the charger used for this article, a Mennekes Amtron, the control unit had ethernet and wireless connectivity, a mobile app, but no public documentation on the protocol.

In cases like that, some options for proceeding are:

• Ask the manufacturer for an API documentation
• Analyze the network communication between management app and charger
• Reverse-engineer a given piece of software, such as a mobile application

Working together, the manufacturer is always the preferred way, and would surely be no issue on our charger. But supposing that the charger is an exotic model without sophisticated customer support, it might happen that obtaining an official API documentation does not succeed.

In this case, analyzing the network traffic between the mobile application and charger would be a least-invasive and promising next step. After setting up a network monitoring environment, i.e. Wireshark and a switch with monitoring port, and listening to the network traffic while using the mobile app, the following findings have been made:

• The charger uses a plaintext HTTP API listening on port 25000
• The API prefix is /MHCP/1.0/
• A client is considered as authenticated when it adds an http query parameter “devKey” with the value of the “Charger PIN1” from the manual
• Request and response payload are JSON-encoded with human readable field names

The listing below shows an example of a state document retrieved from an actual charger on /MHCP/1.0/ChargeData?DevKey=0000:

1{
2  "ChgState": "Idle",
3  "Tariff": "T1",
4  "Price": 280,
5  "Uid": "...",
6  "ChgDuration": 11811,
7  "ChgNrg": 11337,
8  "NrgDemand": 5000,
9  "Solar": 0,
10  "EmTime": 1440,
11  "RemTime": 1440,
12  "ActPwr": 0,
13  "ActCurr": 16,
14  "MaxCurrT1": 16,
15  "BeginH_T1": 4,
16  "BeginM_T1": 30,
17  "PriceT1": 280,
18  "MaxCurrT2": 16,
19  "BeginH_T2": 22,
20  "BeginM_T2": 0,
21  "PriceT2": 200,
22  "RemoteCurr": 16,
23  "SolarPrice": 0,
24  "ExcessNrg": false,
25  "TMaxCurrT1": 16,
26  "TBeginH_T1": 4,
27  "TBeginM_T1": 30,
28  "TPriceT1": 280,
29  "TMaxCurrT2": 16,
30  "TBeginH_T2": 22,
31  "TBeginM_T2": 0,
32  "TPriceT2": 200,
33  "TRemoteCurr": 16,
34  "TSolarPrice": 0,
35  "TExcessNrg": true,
36  "HCCP": "A11"
37}

As the network analysis revealed all details we needed, it was not necessary to reverse-engineer the mobile application. In cases of a less obvious protocol, it is advisable to look into the traffic a bit deeper, as the charger might use an industrial protocol such as Modbus or even some PLC-specific protocol.

Connecting the charger to the cloud

A plain-text http server with cleartext passwords cannot be safely exposed to the internet. Furthermore, having different charger types and interfaces would require our server infrastructure to be aware of multiple different communication protocols and their revisions. To protect the charger from attackers and provide a common protocol for all chargers, we add a tiny edge device which suits as a protocol converter and connectivity provider.

For our use case, this task is done by a Raspberry Pi with a WiFi uplink on the charger’s ethernet port. The protocol converter and forwarding software is a Go binary which:

• periodically polls the HTTP interface from the PLC
• connects to an MQTT server in the cloud
• pushes changed charger state values to the MQTT server

Charger protocol converter

The charger protocol converter’s primary task is to fetch the charger state, map it to a transfer data model and submit it to our cloud.

Fetching the current state from the charger

The code below fetches the current state from the charger via an HTTP call and maps it to a generic charger state struct:

1func (m *MennekesMCP) GetCurrentChargerState() (state models.ChargerState, err error) {
2    resp, errHttp := http.Get(m.changerEndpoint)
3
4    if errHttp != nil {
5        log.WithFields(log.Fields{
6            "error": errHttp,
7        }).Warn("Unable to receive new Reading from charger")
8
9        err = errHttp
10        return
11    }
12
13    var chargeDataPDO MennekesChargeDataPDO
14
15    if jsonDecodeError := json.NewDecoder(resp.Body).Decode(&chargeDataPDO); jsonDecodeError == nil {
16        log.WithFields(log.Fields{
17            "reading": chargeDataPDO,
18        }).Trace("Received new Reading from Charger")
19
20        state = models.ChargerState{
21            OperationState:        mapMennekesState(chargeDataPDO.ChgState),
22            TotalEnergyConsumption: float64(chargeDataPDO.ChgNrg),
23            PowerOutput:            float64(chargeDataPDO.ActPwr),
24            OutputCurrent:          float64(chargeDataPDO.ActCurr),
25            // omitted remaining fields
26        }
27    } else {
28        log.WithFields(log.Fields{
29            "jsonDecodeError": jsonDecodeError,
30        }).Error("Unable to read data from Mennekes Charger")
31        err = jsonDecodeError
32    }
33
34    return
35}

Forwarding charger data

Having obtained data from the charger, the obvious next step is to publish it to our cloud or any other more powerful system for further processing, such as persistence, analysis, or action derivation.

Although using a synchronous HTTP request for that purpose would theoretically work, a message queue appeared as a better fit, because:

• we want to add and remove consumers of the charger data without having to modify the protocol converter or to install an additional gateway
• we expect the next features to require bidirectional communication
• we need a basic “shadow device” functionality, primarily to avoid unnecessary communication with the charger. The message brokers retain feature serves that purpose very well

Because of its smaller footprint compared to AMQP and its popularity in the IoT field, we use MQTT as messaging protocol.

Besides MQTT support in the connected applications, an MQTT broker is required. Depending on the overall setup, this could be a self-managed MQTT broker (such as Mosquitto or HiveMQ ) or a managed broker, i.e. AWS IoT or Azure IoT. In this example, a Mosquitto instance is used.

MQTT topic hierarchy and payload

The MQTT message properties relevant for this case are:

• the topic for routing and delivery purposes
• the QoS statement defining the condition on which a message is considered delivered
• a retain flag, which allows us to declare a message as the current value until a new one is published
• the payload. MQTT is data-agnostic and regards the payload as an array of bytes

We prefer a topic hierarchy based on charger id and variable name over a single topic for all variables for the following reasons:

• It reduces the complexity of a payload, because the topic already indicates what is inside
• It reduces the load on the transport layer, because a fine-grained topic structure allows us to update only the values that have actually changed

Both advantages really come in handy on constrained devices on low bandwidth or expensive connections.

The resulting topic structure is:

chargers/{chargerName as defined}/{valueName as in shared data model}

For example: For a variable “OperationState” on a charger called “carpark1”, the topic is chargers/carpark1/OperationState.

Concerning the payload, a simple solution would be to publish the byte representation of the value (in our case, a 64-bit ANSI float). Supposed that all other applications involved used the same method of floating point representation and don`t care about the time the published value was actually sampled or the latency is extremely low, this might be a viable solution.

In our use case, it is expected that the receiver of a telegram has to be aware of the actual sampling time, for instance for computing integrals over the current power consumption. So we need to include the sampling time into the telegram.

There are several ways to achieve this:

• Put the timestamp into the topic structure, which results in additional messages and increased complexity
• Write the sampling timestamp into an MQTT5 header field
• Instead of the raw value, emit a data structure which contains both timestamp and value

For the given scenario, the third option appears as most reasonable. For the task of encoding a structure into a byte array, one could either use JSON or a data serialization framework, such as AVRO or Google Protocol Buffers. For the sake of simplicity and compatibility with managed solutions (especially their monitoring and debugging tools), we will use plain JSON.

Certificate authentication

MQTT and Mosquitto can be operated with plaintext passwords or even with no passwords at all. In a real-world environment, both edge device and broker should be authenticated and authorized, furthermore it should be possible to isolate device scopes and revoke compromised access credentials.

This requirements are resolved by using certificate-based authentication. The generation and deployment of the certificates might differ depending on the trust infrastructure used in production, for development purposes a self-signed set of certificates might suffice.

After deploying the certificates to the MQTT broker, the Mosquitto broker configuration has to be adapted so that it will only accept MQTT/s connections from trusted devices.

port 8883

cafile /mosquitto/config/certs/rootCA.crt
keyfile /mosquitto/config/certs/server.key
certfile /mosquitto/config/certs/server.crt
tls_version tlsv1.2
require_certificate true
use_identity_as_username true
allow_anonymous false

persistence true
persistence_location /mosquitto/data/

Publishing Charger State

Adding MQTT publisher/subscriber capabilities to a Go application is performed by declaring a dependency to an MQTT library, in our case Eclipse Paho .

After adding an import to github.com/eclipse/paho.mqtt.golang, the project’s go.mod should contain an additional entry: github.com/eclipse/paho.mqtt.golang v1.2.0.

Generating a new MQTT.Client with certificate authentication requires a little more configuration than its plaintext auth counterpart, the following example is inspired by the Paho examples:

1// inspired by
2// https://github.com/eclipse/paho.mqtt.golang/blob/master/cmd/ssl/main.go
3func NewMQTTEmitter(cfg mqtt.Config) (e *MQTTEmitter) {
4    certpool := x509.NewCertPool()
5    pemCerts, err := ioutil.ReadFile(cfg.TrustedCACertificate)
6    if err == nil {
7        certpool.AppendCertsFromPEM(pemCerts)
8    }
9
10    // Import client certificate/key pair
11    cert, err := tls.LoadX509KeyPair(cfg.ClientCertificate, cfg.ClientKey)
12    if err != nil {
13        panic(err)
14    }
15
16    cert.Leaf, err = x509.ParseCertificate(cert.Certificate[0])
17    if err != nil {
18        panic(err)
19    }
20    fmt.Println(cert.Leaf.DNSNames)
21
22    tlsConfig := tls.Config{
23        // RootCAs = certs used to verify server cert.
24        RootCAs: certpool,
25        // ClientAuth = whether to request cert from server.
26        // Since the server is set up for SSL, this happens
27        // anyways.
28        ClientAuth: tls.NoClientCert,
29        // ClientCAs = certs used to validate client cert.
30        ClientCAs: nil,
31        // InsecureSkipVerify = verify that cert contents
32        // match server. IP matches what is in cert etc.
33        InsecureSkipVerify: false,
34        // Certificates = list of certs client sends to server.
35        Certificates: []tls.Certificate{cert},
36    }
37
38    opts := MQTT.NewClientOptions().
39        AddBroker(cfg.Broker).
40        SetTLSConfig(&tlsConfig).
41        SetClientID(cfg.ClientId)
42
43    client := MQTT.NewClient(opts)
44
45    if token := client.Connect(); token.Wait() && token.Error() != nil {
46        panic(token.Error())
47    }
48
49    return &MQTTEmitter{client: client, config: cfg}
50}

As the MQTT connection is established and ready to use, the next step is to put both connection and data acquisition together.

To avoid publishing redundant values, we only publish values that have changed since the last successful publishing operation by implementing a change detection. To keep the change detection short and avoid redundancy, we use reflection to gather a list of properties of the provided charger state and derive a subset consisting of each item which has changed.

1func (e *MQTTEmitter) deriveMessagesFromChangedValues(currentState models.ChargerState, nextState models.ChargerState) (changeMessages []models.ReadyToSendValue) {
2    current := reflect.ValueOf(&currentState).Elem()
3    next := reflect.ValueOf(&nextState).Elem()
4    chargerStateType := current.Type()
5
6    for i := 0; i < chargerStateType.NumField(); i++ {
7        fieldName := chargerStateType.Field(i).Name
8        currentField := current.FieldByName(fieldName)
9        nextField := next.FieldByName(fieldName)
10
11        if changed, currentValue, newValue := hasValueChanged(currentField, nextField); changed {
12            changeMessages = append(changeMessages, models.ReadyToSendValue{
13                Payload: models.ChargerValueEnvelope{
14                    SamplingTime: time.Now(),
15                    Value:        newValue,
16                },
17                Topic: e.config.Topic + fieldName,
18            })
19        }
20    }
21
22    return
23}

The following function publishes the changes to the MQTT bus:

1func (e *MQTTEmitter) EmitChargerState(sampletime time.Time, nextState models.ChargerState) {
2    dueMessages := e.deriveMessagesFromChangedValues(e.previouslyEmittedState, nextState)
3    e.previouslyEmittedState = nextState
4
5    for _, message := range dueMessages {
6        payloadJson, _ := json.Marshal(message.Payload)
7
8        log.WithFields(log.Fields{
9            "sampletime": sampletime,
10            "topic":      message.Topic,
11            "value":      message.Payload.Value,
12        }).Debug("Emitting new State")
13
14        e.client.Publish(message.Topic, 0, retain: true, payloadJson)
15    }
16}

As the publisher flags the messages as retained, the broker can permanently respond to new subscriptions with a “shadow” of the last known values.

Storing data

Having maintained a representation of the current state of the charger, the historic data reported by the chargers should be persisted in a database prior to further processing. Depending on the environment the system is operating in, this could either be a managed solution (such as Azure CosmosDB or AWS RDS) or a self-managed solution. For this tutorial, we use TimescaleDB, a SQL time-series database based on PostgresDB with 100% downwards compatibility.

TimescaleDB / PostgresDB

After installing Timescaledb, we define the schema:

1CREATE TYPE OperationState as enum(
2    'unknown',
3    'off',
4    'idle',
5    'charging',
6    'scheduled_downtime',
7    'unscheduled_downtime'
8);
9
10CREATE TABLE public.ev_readings
11(
12    time TIMESTAMPTZ NOT NULL,
13    OperationState OperationState,
14    TotalEnergyConsumption double precision,
15    PowerOutput double precision,
16    OutputCurrent double precision,
17    MaximumOutputCurrent double precision,
18    ConnectedVehicle varchar(32)
19);

Then, in case of TimescaleDb, we upgrade the table to a Hypertable:

1SELECT create_hypertable('public.ev_readings', 'time')

A Timescale hypertable offers faster time-series queries and additional querying and reporting functionality.

Picking up and storing data from MQTT

Reading new values from the MQTT bus is done by setting up a subscription, consisting of the topic (or expression) to listen to and the handler function.

The implementation below subscribes to every topic below the chargerRootTopic, such as chargers/carpark1 and passes HandleNewMessage as handler function.

1subscriber := &MQTTSubscriber{
2        client:  client,
3        config:  cfg,
4        handler: HandleNewMessage,
5        context: ctx}
6
7subscriber.chargerRootTopic = chargerRootTopic + "/#"
8subscriber.client.Subscribe(chargerRootTopic, 0, subscriber.HandleNewMessage)

The handling function has to deserialize the message and act as an adapter to the actual domain specific handler:

1func (m *MQTTSubscriber) HandleNewMessage(client MQTT.Client, message MQTT.Message) {
2    var chargerReading models.ChargerValueEnvelope
3    messageReader := bytes.NewReader(message.Payload())
4
5    if errDecodeJson := json.NewDecoder(messageReader).Decode(&chargerReading); errDecodeJson == nil {
6        readingName := strings.Replace(message.Topic(), m.chargerRootTopic+"/", "", -1)
7
8        log.WithFields(log.Fields{
9            "charger": message.Topic(),
10            "msg":     message.MessageID(),
11            "reading": chargerReading,
12        }).Trace("Received new Reading from Charger")
13
14        m.handler.Handle(readingName, chargerReading)
15
16    } else {
17        log.Warn("Unable to decode incoming state json telegram")
18    }
19}

For the scope of this article, we just save the acquired data to the database:

1func (p *Persistor) Handle(readingName string, reading models.ChargerValueEnvelope) error {
2    log.WithFields(log.Fields{
3        "readingName": readingName,
4        "value" : reading.Value,
5    }).Debug("Handling new charger message")
6
7    conn, err := p.database.Acquire(context.Background())
8
9    if err != nil {
10        log.WithFields(log.Fields{
11            "error": err,
12        }).Warn("Error while connecting to db")
13        return err
14    }
15    defer conn.Release()
16
17    sanitizer := regexp.MustCompile(`[^a-zA-Z]`)
18    sanitizedReadingName := sanitizer.ReplaceAll([]byte(readingName), []byte(""))
19
20    sql := fmt.Sprintf("INSERT INTO public.ev_readings " +
21        "(time, %s) VALUES ($1, $2)", string(sanitizedReadingName))
22
23    samplingTimeUtc := reading.SamplingTime.UTC()
24    _, errInsert := conn.Exec(context.Background(), sql, samplingTimeUtc, reading.Value)
25
26    if errInsert != nil {
27        log.WithFields(log.Fields{
28            "error": errInsert,
29        }).Warn("Error while inserting data")
30        return errInsert
31    }
32
33    return nil
34}

After running both data acquisition and persistor while a vehicle is connected and charging, the database table should look similar to the table below:

Summary & next steps

This tutorial covered basic concepts of MQTT messaging design for an electric vehicle charger infrastructure, protocol analysis on unknown devices, database setup and putting it all together with the Go programming language.

Having a constant stream of updated state messages and a database with the entire history, this setup can act as a starting point for extensions such as dashboards or rule engines.