Logo der codecentric AG, einem in Deutschland führenden IT-Consulting Unternehmen

Logo der codecentric AG einem in Deutschland führenden IT-Consulting Unternehmen

Cloud-Native

Data & Machine Learning

Generative KI

IT-Security

Produktentwicklung

Weitere Leistungen...

Leistungen

Arbeitsweise

Projekte

Berlin

Bielefeld

Dortmund

Erfurt

Frankfurt am Main

Hamburg

Karlsruhe

Leipzig

München

Münster

Nürnberg

Solingen

Stuttgart

Standorte

Blog

Softwerker

Fachmedien

Webinare

Events

Wissens-Hub

Unsere Kultur

Kultur & Karriere Podcast

Deine Entwicklung

Jobs

Karriere

Über uns

Aktuelles

Kontakt

Alle Standorte

Du hast Fragen oder möchtest dich mit mir austauschen? Gern – trag einfach deine Kontaktdaten ein.

Contact Us | Sub-Navi-Kontakt oder Zero Search Hits

Kontakt aufnehmen

Marketing & Community Management

Konntest du keine Einträge zu deinem Thema finden?  

STANDORT-TEAM

Unser codecentric Team am Standort Karlsruhe stellt sich vor.

codecentric Team Karlsruhe

Willkommen im Team Karlsruhe

<p>Das Interesse an der Programmiersprache Kotlin wächst, und auch die Verwendung von Kotlin in Projekten nimmt zu. Ein Bereich, in dem Kotlin hervorragend verwendet werden kann, ist die Implementierung von speziellen Domänen-spezifischen Sprachen, den DSLs.<br /> Der <a href="https://de.wikipedia.org/wiki/Dom%C3%A4nenspezifische_Sprache">Wikipedia-Artikel</a>  definiert:</p><blockquote><p>Eine domänenspezifische Sprache … oder anwendungsspezifische Sprache ist eine formale Sprache, die zur Interaktion zwischen Menschen und digital arbeitenden Computern („Computersprache“) für ein bestimmtes Problemfeld (die sogenannte Domäne) entworfen und implementiert wird.</p></blockquote><p>In diesem Artikel werde ich eine minimale DSL für den Zugriff auf Apache Kafka implementieren. Diese Sprache verwendet Schlüsselwörter wie <em>kafka, producer, consumer.</em> Sie definiert ebenso eine hierarchische Struktur, welche die Beziehungen dieser Bergiffe modelliert.</p><p>Der Zweck dieses Blog-Posts ist es, zu zeigen, wie eine DSL mit Kotlin erstellt werden kann. Die erzeugte DSL wird nur eine minimale Funktionalität haben. Die Nachrichten, die in Kafka Topics geschrieben werden, sind reine Strings. Der Fokus des Artikels liegt auf dem Schreiben der DSL, nicht auf der Benutzung von Apache Kafka.</p><p>Der komplette Code für dieses Beispiel ist auf <a href="https://github.com/pjmeisch-cc/kotlin-kafka-dsl">Github</a>  verfügbar.</p><h2>Die DSL</h2><p>Die beiden folgenden Code-Stücke zeigen die Verwendung der DSL. Das erste Beispiel zeigt einen Producer, der Strings in ein Kafka Topic schreibt. Das zweite Beispiel ist eine Consumer-Anwendung, welche aus Kafka liest.</p><p>Ich zeige zuerst den fertigen Code bevor ich erkläre, wie man diese DSL erzeugt:</p><h3>Der Producer</h3>

```
fun main(args: Array) {
    val log: Logger = LoggerFactory.getLogger("de.codecentric.ProducerRunnerDSL")
    
    kafka("localhost:9092") {
        producer("kt-topic") {
            (1..10).forEach {
                val msg = "test message $it ${LocalDateTime.now()}"
                log.info("sending $msg")
                send(msg)
            }
            flush()
        }
    }
}

```

<p>Ich verwende als erstes die <code>kafka("localhost:9092")</code>-Methode, um die Verbindungsdetails zum Kafka Broker zu definieren, der auf meinem lokalen Rechner läuft. Innerhalb dieses Kontextes erzeuge ich dann einen Producer mit <code>producer("kt-topic")</code>.</p><p>Im Kontext des Producers ist eine Schleife, welche 10 Nachrichten erzeugt. Diese Nachrichten werden durch den Aufruf der Producermethode <code>send(msg)</code> an Kafka geschickt. Nach der Schleife sorgt der Aufruf von <code>flush()</code> dafür, dass alle Nachrichten an Kafka geschickt wurden, bevor das Programm terminiert.</p><h3>Der Consumer</h3>

```
fun main(args: Array) {
  val log: Logger = LoggerFactory.getLogger("de.codecentric.ConsumerRunnerDSL")

  kafka("localhost:9092") {
    consumer("kt-topic") {
      Runtime.getRuntime().addShutdownHook(Thread(Runnable {
        stop()
      }))
      consume {
        log.info("got $it")
      }
    }
  }
}

```

<p>Wie der Producer definiert auch die Consumer-Anwendung zuerst die Verbindung zum Kafka Broker. Sie erzeugt dann einen Consumer für das topic mit <code>consumer("kt-topic")</code>. In diesem Consumer wird ein Shutdownhook installiert, der sicherstellt, dass die Methode <code>stop()</code> des Consumers am Programmende aufgerufen wird, damit der intern verwendete <code>KafkaConsumer</code>korrekt beendet wird.</p><p>Danach wird die Methode <code>consume</code> des Consumers aufgerufen mit einer Lambda-Funktion als Parameter, welche einfach den String protokolliert, der aus dem Kafka Topic gelesen wurde. Die <code>consume</code>-Methode blockiert, während sie Daten aus Kafka liest und kehrt nur bei einem Fehler zurück oder wenn die Methode <code>stop()</code> von einem anderen Thread aus aufgerufen wird, zum Beispiel beim Programmende.</p><h2>Voraussetzungen</h2><h3>Apache Kafka</h3><p>Um die Programme laufen zu lassen benötigt man <a href="https://kafka.apache.org/">Apache Kafka</a> . Ich verwende <a href="https://docs.confluent.io/current/installation/installing_cp.html">Confluent Platform OpenSource</a> , um meine Implementierung zu testen, aber jedes andere Apache Kafka Setup ist auch in Ordnung.</p><h3>Kotlin-Konzepte</h3><p>Zwei Konzepte der Sprache Kotlin sind essentiell, um eine DSL in der Form zu schreiben, wie ich es in diesem Artikel zeige:</p><p><strong>Lambdas als Funktionsparameter</strong><br /> Wenn eine Funktion als letzten Aufrufparameter eine andere Funktion erwartet, und dieser Parameter wird in Form eines Lambda übergeben, dann schreibt man das Aufrufargument <strong>hinter</strong> hinter die Klammern mit den anderen Parametern (<a href="https://kotlinlang.org/docs/reference/lambdas.html#passing-a-lambda-to-the-last-parameter">Dokumentation</a> ):</p>

```
// func hat einen String als erstes und eine function von String nach String als zweites Argument.
// func ruft die als Argument übergebene Funktion mit dem ersten Parameter auf und gibt den Rückgabewert aus
fun func(arg1: String, calledWithArg1: (String) -> String) {
  println(calledWithArg1(arg1))
}

// man kann den Aufruf so schreiben:
func("foo", { s: String -> s + s })
// Ausgabe ist foofoo

// in Kotlin besser so:
func("foo") {
  it + it
}
//Ausgabe ist foofoo

```

<p><strong>extension functions:</strong><br /> Extension Functions erlauben es, Funktionalität zu einer vorhandenen Klasse hinzuzufügen. Dabei werden sie nicht in der Klasse selber definiert (<a href="https://kotlinlang.org/docs/reference/extensions.html#extension-functions">Documentation)</a>  – das Beispiel hier ist etwas off-topic:</p>

```
class PaymentInfo(var amount: Double, var creditCard: String)

val p = PaymentInfo(42.0, "1234-5678-0123-1234")

// extension function um den Preis zu verdoppeln, nicht in der Klasse definiert
fun PaymentInfo.twice() {
  // hier ist 'this' das PaymentInfo Objekt
  amount *= 2
}

// die Funktion wird wie eine Methode der Klasse selbst aufgerufen.
p.twice()
// der Betrag ist jetzt 84.0

// eine Function um ein PaymentInfo zu modifizieren, dabei ist der modifier eine extension function
// die als Parameter übergeben wird
fun modifyPayment(payment: PaymentInfo, modifier: PaymentInfo.() -> Unit) {
  payment.modifier()
}

// maskieren der Kreditkartennummer
modifyPayment(p) {
  // hier ist 'this' das PaymentInfo Objekt
  creditCard = "XXX"
}
// im PaymentInfo Objekt ist jetzt die Kreditkarte maskiert

```

<p>Wir können extension functions zum einen verwenden, um Funktionalität zu Klassen hinzuzufügen, die nicht selber modifiziert werden können, wie zum Beispiel <code>String</code> oder <code>List</code>.<br /> Zum anderen bieten extension functions die Möglichkeit, einer Funktion ein Lambda mitzugeben, welches dann im Kontext des erweiterten Objektes ausgeführt wird. Im Beispielcode wird der <code>modifyPayment()</code>-Methode eine extension function als Parameter übergeben. Diese wird im Kotlin Stil hinter den Methodenaufruf geschrieben.</p><h2>Schritt 1: interne Klassen um auf Apache Kafka zuzugreifen</h2><p>In einem ersten Schritt schreiben wir drei Klassen, um in Kafka zu schreiben und von Kafka zu lesen. Diese Klassen verwenden die offizielle Kafka Client Library, die in Java geschrieben ist. Sie bilden eine Kapselung der Kafka-Bibliothek in Kotlin.</p><h3>Kafka-Konfigurationsklasse</h3>

```
data class Kafka(val bootstrapServers: String)

```

<p>Die erste Klasse dient der Speicherung der Konfiguration für die Verbindung zu Kafka. Die konfigurierbaren Eigenschaften sind hier auf ein Minimum reduziert.</p><h3>Die Producer-Klasse</h3>

```
class Producer(kafka: Kafka, private val topic: String) {
  private val kafkaProducer: KafkaProducer<String, String>

  init {
    val config = Properties()
    config[ProducerConfig.BOOTSTRAP_SERVERS_CONFIG] = kafka.bootstrapServers
    config[ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java
    config[ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java
    kafkaProducer = KafkaProducer(config)
  }

  fun send(msg: String) {
    kafkaProducer.send(ProducerRecord(topic, msg))
  }

  fun flush() = kafkaProducer.flush()
}

```

<p>Die Producer-Klasse wird mit einem <code>Kafka</code>-Konfigurationsobjekt und dem Namen des Topic initialisiert. Im <code>init</code>-Block wird ein <code>KafkaProducer</code> aus der offiziellen Client-Bibliothek mit den notwendigen Konfigurationswerten angelegt. Die <code>send(msg: String)</code>-Methode erzeugt einen <code>ProducerRecord</code> und sendet ihn zu Kafka, und die <code>flush()</code> Methode wird an die Client-Bibliothek durchgereicht.</p><h3>Die Consumer-Klasse</h3>

```
class Consumer(kafka: Kafka, topic: String) {
  private val kafkaConsumer: KafkaConsumer<String, String>

  @Volatile
  var keepGoing = true

  init {
    val config = Properties()
    config[ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG] = kafka.bootstrapServers
    config[ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
    config[ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
    config[ConsumerConfig.GROUP_ID_CONFIG] = UUID.randomUUID().toString()
    kafkaConsumer = KafkaConsumer<String, String>(config).apply {
      subscribe(listOf(topic))
    }
  }

  fun consume(handler: (value: String) -> Unit) = Thread(Runnable {
    keepGoing = true
    kafkaConsumer.use { kc ->
      while (keepGoing) {
        kc.poll(500)?.forEach {
          handler(it?.value() ?: "???")
        }
      }
    }
  }).start()

  fun stop() {
  keepGoing = false
  }
}

```

<p>Auch diese Klasse wird mit der Kafka-Konfiguration und dem Topic initialisiert und erzeugt eine <code>KafkaConsumer</code>-Instanz, die sich beim gewünschten Topic registriert. Die Klasse hat eine Property <code>keepGoing</code>, welche bei Aufruf der Methode <code>consume(handler: (value: String) -> Unit)</code> auf true gesetzt wird. Diese Methode startet einen Thread, in dem das Kafka Topic gepolled wird und in dem für jeden empfangenen Wert die Handler-Methode aufgerufen wird.</p><p>Die <code>stop()</code>-Methode wird verwendet, um das <code>keepRunning</code>-Flag auf false zu setzen, was dazu führt, dass das Polling und damit auch der Thread beendet wird.</p><h3>Beispielanwendungen für diese Klassen</h3><p>Als Beispiel hier zwei Klassen, welche diesen Code verwenden. Hier wird noch keine DSL verwendet, es handelt sich um einfachen Kotlin Code:</p>

```
fun main(args: Array) {
  val log: Logger = LoggerFactory.getLogger("de.codecentric.ProducerRunner")

  val kafka = Kafka("localhost:9092")
  val topic = "kt-topic"

  val producer = Producer(kafka, topic)
  (1..10).forEach {
    val msg = "test message $it ${LocalDateTime.now()}"
    log.info("sending $msg")
    producer.send(msg)
  }
  producer.flush()
}

```

<p> </p>

```
fun main(args: Array) {
  val log: Logger = LoggerFactory.getLogger("de.codecentric.ConsumerRunner")

  val kafka = Kafka("localhost:9092")
  val topic = "kt-topic"

  val consumer = Consumer(kafka, topic)
  Runtime.getRuntime().addShutdownHook(Thread(Runnable {
    consumer.stop()
  }))
  consumer.consume {
    log.info("got $it")
  }
}

```

<h2>Schritt 2: der Code für die DSL</h2><p>Um die DSL zu schreiben, welche die gerade definierten Klassen verwendet, wird der folgende Code benötigt:</p>

```
class KafkaDSL(bootstrapServers: String) {
  private val kafka = Kafka(bootstrapServers)

  fun producer(topic: String, doProduce: Producer.() -> Unit) =
    Producer(kafka, topic).doProduce()

  fun consumer(topic: String, doConsume: Consumer.() -> Unit) =
    Consumer(kafka, topic).doConsume()
}

fun kafka(bootstrapServers: String, init: KafkaDSL.() -> Unit) =
  KafkaDSL(bootstrapServers).init()

```

<p>Mehr ist nicht notwendig. Ich werde die Klassen und Funktionen Schritt für Schritt erläutern an Hand des Producer und Consumer Beispiels vom Anfang des Artikels:</p><h3>Der Producer</h3>

```
kafka("localhost:9092") {
  // Dies ist der Rumpf der KafkaDSL.init()-Funktion.
  // In diesem Block ist 'this' das KafkaDSL-Objekt.
}

```

<p>Dieser Aufruf der globalen <code>kafka</code>-Funktion erzeugt ein <code>KafkaDSL</code>-Objekt und übergibt die bootstrap Server im Konstruktor. Das <code>KafkaDSL</code>-Objekt verwendet diese Information, um ein <code>Kafka</code>-Konfigurationsobjekt zu erstellen. Das zweite Argument – <code>init</code> – der <code>kafka()</code>-Methode ist eine Kotlin Extension für die <code>KafkaDSL</code>-Klasse, die – im Kotlin-Stil – hinter die <code>kafka()</code>-Klammern geschrieben wird. Diese Funktion wird für das neu erzeugte <code>KafkaDSL</code>-Objekt aufgerufen.</p>

```
kafka("localhost:9092") {
  // Dies ist der Rumpf der KafkaDSL.init()-Funktion.
  // In diesem Block ist 'this' das KafkaDSL-Objekt.
  producer("kt-topic") {
    // Dies ist der Rumpf der Producer.doProduce()-Funktion.
    // In diesem Block ist 'this' das Producer-Objekt.
  }
}

```

<p>Die <code>KafkaDSL.init()</code>-Funktion, die für das <code>KafkaDSL</code>-Objekt übergeben wurde, ruft die Methode <code>producer("kt-topic"){}</code> auf. Diese Methode ist in der <code>KafkaDSL</code>-Klasse spezifizert, der erste Parameter ist der Name des Topic und der zweite ist eine Extension-Funktion für die <code>Producer</code>-Klasse. Dieser Aufruf erzeugt ein Producer-Objekt mit den notwendigen Argumenten und ruft dann die Extension-Funktion für dieses Objekt auf.</p>

```
kafka("localhost:9092") {
  // Dies ist der Rumpf der KafkaDSL.init()-Funktion.
  // In diesem Block ist 'this' das KafkaDSL-Objekt.
  producer("kt-topic") {
    // Dies ist der Rumpf der Producer.doProduce()-Funktion.
    // In diesem Block ist 'this' das Producer-Objekt.
    (1..10).forEach {
      send("test message $it ${LocalDateTime.now()}") // <- Producer.send()
    }
    flush() // <- Producer.flush()
  }
}

```

<p>Im Rumpf der <code>Producer.doProduce()</code>-Extension-Funktion ist die Schleife implementiert, welche die Nachrichten erzeugt. Diese Nachrichten werden mit den <code>send()</code>– und <code>flush()</code>-Methoden des <code>Producer</code>-Objekts versandt.</p><h3>Der Consumer</h3><p>Der Code für den Consumer entspricht dem Code für den producer, ich werde nur die Unterschiede erläutern. Zuerst wird ein <code>KafkaDSL</code>-Objekt erzeugt. Der Unterschied liegt in der Implementierung der <code>KafkaDSL.init()</code>-Funktion:</p>

```
kafka("localhost:9092") {
  // Dies ist der Rumpf der KafkaDSL.init()-Funktion.
  // In diesem Block ist 'this' das KafkaDSL-Objekt.
  consumer("kt-topic") {
    // Dies ist der Rumpf der Consumer.doConsume()-Funktion.
    // In diesem Block ist 'this' das Consumer-Objekt.
  }
}

```

<p>Hier wird die <code>consumer("kt-topic")</code>-Methode aufgerufen, die ein <code>Consumer</code>-Objekt erzeugt und dann die <code>Consumer.doConsume()</code>-Extension-Funktion aufruft, welche als zweiter Parameter übergeben wurde.</p>

```
kafka("localhost:9092") {
  // Dies ist der Rumpf der KafkaDSL.init()-Funktion.
  // In diesem Block ist 'this' das KafkaDSL-Objekt.
  consumer("kt-topic") {
    // Dies ist der Rumpf der Consumer.doConsume()-Funktion.
    // In diesem Block ist 'this' das Consumer-Objekt.
    Runtime.getRuntime().addShutdownHook(Thread(Runnable {
      stop() // <- Consumer.stop()
    }))
    consume { // <- Consumer.consume(handler)
      log.info("got $it")
    }
  }
}

```

<p>In der <code>Consumer.doConsume()</code>-Funktion wird zuerst der Shutdownhook installiert und dann die Methode <code>consume()</code> des <code>Consumer</code> ausfgerufen. Dieser Methode wird ein Lambda übergeben als Handler, um die aus Kafka gelesenen Wert zur verarbeiten.</p><p>Mehr benötigen wir nicht, umn unsere kleine DSL zu schreiben.</p><h2>Fazit</h2><p>Zuerst haben wir ein paar Hilfsklassen geschrieben für die gewünschte Funktionalität zum Lesen und Schreiben in und aus Kafka. Und mit nur einer globalen Funktion und einer Hilfsklasse mit zwei Methoden haben wir unsere DSL definiert. Das Beispiel zeigt das Potential, das in Kotlins Sprachkonzepten wie Extension-Funktionen oder Lambda-Argumenten steckt. Weitere Möglichkeiten sind noch Operator-Overloading und Infix-Funktionen, die ich hier nicht verwendet habe, um das Beispiel nicht zu komplex zu machen.<br /> Für Java-Entwickler sind diese Spracheigenschaften zuerst ungewohnt, aber es lohnt sich, sich diese genauer anzuschauen.</p><p>Ich hoffe, euch hat dieser Ausflug in die Welt von Kotlin gefallen.</p>

Wie schreibt man eine Kotlin-DSL – z.B. für Apache Kafka?

<p>In diesem Beitrag möchte ich einige Features von Kotlin vorstellen, die es ermöglichen, kurzen und präzisen Code zu schreiben. Zum einen sind dies Funktionen der Kotlin-Standard-Bibliothek, zum anderen eigene Funktionen, welche Eigenschaften der Sprache Kotlin verwenden wie zum Beispiel Extension Functions und Lambda-Argumente.</p><p>Ich gehe nicht im Detail darauf ein, was Extension Functions sind oder auf spezielle Syntaxaspekte. Hierfür möchte ich auf die hervorragende <a href="http://kotlinlang.org/docs/reference" target="_blank">Kotlin Documentation verweisen</a> .</p><h2>Funktionen aus der Kotlin-Standard-Bibliothek</h2><p>Im ersten Teil des Artikels möchte ich einige Funktionen der Kotlin-Standard-Bibliothek vorstellen. Es handelt sich hier um eine kleine Auswahl, die ich nützlich finde, und ich möchte den Leser ausdrücklich ermutigen, sich den Quellcode dieser Funktionen anzuschauen und die <a href="http://kotlinlang.org/api/latest/jvm/stdlib/index.html" target="_blank">Dokumentation</a>  zu lesen.</p><h3>Überprüfen von Argumenten</h3><p>Wenn man in seinem Code kein Framework zur Validierung verwendet, ist es oft notwendig, die Gültigkeit von Argumenten zu überprüfen. Im Normalfall wird das dann wie im folgenden Beispiel gemacht:</p>

```
fun someFunction(answer: Int) {
  if(answer != 42) {
    throw IllegalArgumentException("the answer must be 42!")
  }
  // function code
}
```

<p>Besser ist es, die <code>require</code>-Funktion der Standard-Bibliothek zu verwenden, welche die gleiche Funktionalität bietet, aber weniger Code benötigt:</p>

```
fun someFunction(answer: Int) {
  require(answer == 42) {" the answer must be 42!"}
  // function code
}
```

<h3>Mehrere Werte aus einer Funktion zurückgeben</h3><p>Manchmal ist es nötig, dass eine Funktion zwei Werte zurückgibt. Eine Möglichkeit hierfür ist es, eine neue Klasse anzulegen, die diese Werte enthält. Einfacher ist es, die <code>Pair</code>-Klasse aus Kotlin zu verwenden, zusammen mit der <code>to</code> infix-Funktion und einer destructuring-Variablendeklaration:</p>

```
fun someFunction() : Pair<Int, String> {
  return 42 to "the answer"
}

fun main(args: Array<String>) {
  val (i: Int, s: String) = someFunction()
  log.info("$s is $i") // logs "the answer is 42"
}
```

<h3>Zeitmessungen</h3><p>Um die Ausführungszeit von Programmteilen zu messen, wird häufig Code in dieser Form verwendet:</p>

```

val start = System.currentTimeMillis()
// do something    
val end = System.currentTimeMillis()   

log.info("duration: ${end - start} msecs")

```

<p>Kotlin bietet hierfür zwei inline-Funktionen, <code>measureTimeMillis</code> und <code>measureNanoTime</code>, die folgendermassen verwendet werden:</p>

```

val duration = measureTimeMillis {
  // do something
}

log.info("duration: $duration msecs")
```

<h3>File IO</h3><p>Die normalen JVM-Bibliotheken bieten keine Funktionen um ein Verzeichnis zu löschen, das nicht leer ist. Hierfür ist es notwendig, die im Verzeichnis liegenden Dateien und Unterverzeichnisse rekursiv zu löschen.<br /> Kotlin hat für diesen Zweck eine Extension Function:</p>

```
File("/path/to/some/directory").deleteRecursively()
```

<p>Falls man den Namen einer Datei ohne die Dateiendung benötigt:</p>

```

val f = File("image.jpg")
val name = f.nameWithoutExtension // sets name to "image"

```

<p>Eine Datei zeilenweise einlesen und für jede Zeile eine Funktion aufrufen:</p>

```

File("filename").useLines { line ->
  println(line)
}

```

<p>Es gibt noch sehr viel mehr nützliche Funktionen im Package <code>kotlin.io</code>, auch hier verweise ich auf die Dokumentation.</p><h3>Nebenläufige Programmierung</h3><h4>Reentrant read write locks</h4><p>Für den Fall, dass man eine Resource absichern möchte, so dass immer nur ein Thread Schreibzugriff hat, aber mehrere Threads Lesezugriff, gibt es in der Java-Runtime die Klasse <code>ReentrantReadWriteLock</code>. Die korrekte Verwendung dieser Klasse ist aber nicht trivial, da auf Interruptions oder Exceptions geachtet werden muss, wenn die entsprechenden Locks belegt und freigegeben werden.</p><p>Kotlin bietet einige Extension-Functions der <code>ReentrantReadWriteLock</code> Klasse an, mit denen die Verwendung sehr vereinfacht wird:</p>

```
val sharedResource = mutableMapOf<String, String>()
val lock = ReentrantReadWriteLock()

fun readSomeData(key: String) {
  return lock.read {
    sharedResource[key]
  }
}

fun writeSomeData(key: String, value: String) {
  lock.write {
    sharedResource[key] = value
  }
}
```

<h4>Threads und ThreadLocal Objekte</h4><p>Wenn man ein <code>ThreadLocal</code>-Objekt verwendet, wird es normalerweise wie folgt initialisiert:</p>

```
class ClassWithThreadState {
  private val state = object : ThreadLocal<String>() {
    override fun initialValue(): String = "initialValue"
  }

  fun someFunction() {
    var currentState = state.get()
    log.info { "currentState: $currentState" }
    state.set("newState")
  }
}
```

<p>Kotlin fügt der Klasse die Funktion <code>getOrSet</code> hinzu, welche die Initiaisierung einfacher macht:</p>

```
class ClassWithThreadState {
  private val state = ThreadLocal<String>()

  fun someFunction() {
    val currentState = state.getOrSet { "initalValue" }
    log.info { "currentState: $currentState" }
    state.set("newState")
  }
}
```

<p>Um Code in einem eigenen Thread auszuführen, muss man ihn in ein <code>Runnable</code> einpacken, welches dann einem <code>Thread</code>-Objekt übergeben wird, das dann wiederum gestartet wird:</p>

```
fun main(args: Array<String>) {
  val classWithThreadState = ClassWithThreadState()
  Thread(Runnable { classWithThreadState.someFunction() }).start()
}
```

<p>Die Kotlin-Bibliothek bietet eine Funktion <code>thread</code>, die das Ganze vereinfacht:</p>

```
fun main(args: Array<String>) {
  val classWithThreadState = ClassWithThreadState()
  // imediately start the thread with the runnable
  thread { classWithThreadState.someFunction() }
    
  // or first create it and start it later:
  val t = thread(start = false) { classWithThreadState.someFunction() }
  t.start()
}
```

<p>Die <code>thread</code>-Funktion hat neben dem <code>start</code>-Parameter weitere optionale Parameter, die es erlauben, zum Beispiel die Priorität, den Namen oder auch den Daemon-Status des erzeugten Thread zu setzen.</p><h2>Eigene (Extension-)Funktionen</h2><p>Im zweiten Teil diese Artikels möchte ich zeigen, wie leicht es ist, eigene Funktionen zu schreiben, die dazu beitragen, dass der Code präziser, einfacher und besser zu warten wird.</p><p>Die Prinzipien, die ich für diese Funktionen verwende, sind die gleichen, die in der Kotlin-Standard-Bibliothek verwendet werden: Extension-Funktionen und Funktionen, die ein Lambda als letzten Parameter nehmen. Für weitere Informationen zu diesen Techniken verweise ich auf meinen früheren Blogpost <a href="/wissens-hub/blog/kotlin-dsl-apache-kafka">“ Wie schreibt man eine Kotlin-DSL“</a> .</p><h3>Slf4j</h3><p>Jeder Entwickler, der <em>slf4j</em> verwendet, kennt den Code um einen <code>Logger</code> zu erzeugen, bei dem der Name der Klasse wiederholt werden muss:</p>

```
class SomeClass {
  fun doSomething() {
    log.info("doing something")
  }
    
  companion object {
    // always have to repeat the name of the class here:
    private val log: Logger = LoggerFactory.getLogger(SomeClass::class.java)
  }
}
```

<p>Mit einer kleinen Extension-Funktion für die <code>Any</code>-Klasse ist es möglich, einen <em>slf4j</em>-Logger für jede Klasse zu erzeugen, auch wenn der Logger als Property eines companion-Objekt definiert wird:</p>

```
fun Any.logger(): Logger {
  val clazz = if (this::class.isCompanion) this::class.java.enclosingClass else this::class.java
  return LoggerFactory.getLogger(clazz)
}
```

<p>Bei der Verwendung dieser Funktion ist es nicht mehr nötig, die Klasse, für die der Logger erzeugt wird, anzugeben:</p>

```

package de.codecentric.kotlingoodies
import logger

class SomeClass {
  fun doSomething() {
    log.info("doing something")
  }

  companion object {
    // creates a logger with the name "de.codecentric.kotlingoodies.SomeClass"
    private val log = logger()
  }
}
```

<p>Wenn man zum Erstellen in der Log-Message Aufrufe wie String-Templating verwendet, empfiehlt es sich, vorher das Level des Loggers zu prüfen:</p>

```
fun someFunction() {
  val someObject = Any() // or something else, doesn't matter here

  try {
    someObject.doSomething()
    if(log.isDebugEnabled) {
      log.debug("doing something with $someObject")
    }
  } catch (e: Exception) {
    if(log.isWarnEnabled) {
      log.warn("could not do something with $someObject" e)
    }
  }
}
```

<p>Durch ein paar Extension-Funktionen der <code>Logger</code>-Klasse kann dies erheblich vereinfacht werden – ich zeige hier nur die Funktionen für das <em>debug</em>– und <em>info</em>-Level:</p>

```
inline fun Logger.info(msg: (() -> String)) {if (isInfoEnabled) info(msg())}
inline fun Logger.info(t: Throwable, msg: (() -> String)) {if (isInfoEnabled) info(msg(), t)}
inline fun Logger.debug(msg: (() -> String)) {if (isDebugEnabled) debug(msg())}
inline fun Logger.debug(t: Throwable, msg: (() -> String)) {if (isDebugEnabled) debug(msg(), t)}

// use it like this: 
fun someFunction() {
  val someObject = Any() // or something else, doesn't matter here

  try {
    someObject.doSomething()
    log.debug { "doing something with $someObject" }
  } catch (e: Exception) {
    log.warn(e) { "could not do something with $someObject" }
  }
}
```

<p>Diese Funktionen machen den Code erheblich lesbarer und verhindern auch, dass man das Loglevel für den Aufruf ändert und dabei vergisst, die dazugehörige Abfrage an das neue Level anzupassen.</p><h3>Funktionen, um Boilerplate-Code zu kapseln</h3><p>Es kommt häufig vor, dass im Code immer das Gleiche gemacht wird: zum Beispiel eine Transaktion starten, etwas ausführen, im Erfolgsfall die Transaktion committen und im Fehlerfall ein Rollback durchführen.</p><p>Um das elegant und wiederverwendbar zu schreiben, definiert man eine Funktion – inline und reified um den Rückgabewert zu erhalten – und übergibt dieser Funktion ein Lambda mit dem auszuführenden Code. Hier ein Beispiel mit einer Funktion <em>transactional</em>:</p>

```
inline fun <reified R> transactional(f: () -> R): R {
    // code to begin transaction
  return try {
    f()
    // code to commit transaction
  } catch(e: Exception){
    // code to rollback transaction
    throw e
  }
}
```

<p>Verwendet wird die Funktion dann wie folgt:</p>

```
data class Record(val name: String)
fun main(args: Array<String>) {

  val records = transactional { 
    // this code is run within a transaction and is probably more complex in a real world scenario
    listOf(Record("John"), Record("James"))
  }

  records.forEach { println(it) }
}
```

<h3>Diakritische Zeichen entfernen</h3><p>Kürzlich habe ich eine Funktion benötigt, die aus einem String diaktritische Zeichen (z. B. Akzente) entfernt. Ein erster Ansatz hierfür sieht folgendermassen aus:</p>

```
fun noDiacritics (s: String) : String {
  val normalized = Normalizer.normalize(this, Normalizer.Form.NFD)
  val stripped = normalized.replace("\\p{M}".toRegex(), "")
  return stripped
}
```

<p>Eleganter und besser ist es, diese Logik in eine Extension-Funktion der <code>String</code>-Klasse zu legen, damit kann sie für jede String-Variable und auch für String-Konstanten aufgerufen werden:</p>

```

inline fun String?.noDiacritics() = this?.let{Normalizer.normalize(this, Normalizer.Form.NFD)
  .replace("\\p{M}".toRegex(), "")}

fun main(args: Array<String>) {
  println("Porsche Coupé".noDiacritics())
  // output is: Porsche Coupe
}
```

<h2>Zusammenfassung</h2><p>In diesem Artikel habe ich gezeigt, wie kleine Funktionen – entweder aus der Kotlin-Standard-Bibliothek oder selbst geschrieben – dazu führen können, dass der eigene Code kürzer, besser lesbar und besser wartbar wird.</p><p>Kotlin-Eigenschaften wie Extension-Funktionen, inline- oder infix-Funktionen helfen dabei, solche „Hilfs“funktionen zu schreiben.</p><p>Ich kann nur empfehlen, sich den Quellcode der Kotlin-Standard-Bibliothek anzuschauen, man kann hier immer wieder Neues entdecken und lernen.</p>

Kotlin-Leckerbissen, die dem Entwickler das Leben erleichtern

The Kotlin language is gaining more and more attention and is being used in an increasing number of projects. One thing that Kotlin can be used for is implementing special domain-specific-languages (DSLs).<br /> The <a href="https://de.wikipedia.org/wiki/Dom%C3%A4nenspezifische_Sprache">Wikipedia entry on DSL</a>  states:</p><blockquote><p>A domain-specific language (DSL) is a computer language specialized to a particular application domain.</p></blockquote><p>In this post I will implement a minimal DSL for accessing Apache Kafka which uses keywords like <em>kafka, producer, consumer.</em> It also defines a nesting structure that models the relationship between these concepts.</p><p>The purpose of this blog post is to show how to create a custom DSL with Kotlin. The DSL created will only have minimal functionality. The messages that are written to the Kafka topics are just plain strings. The focus is on creating the DSL, not using Apache Kafka.</p><p>The complete code for this sample project is available on <a href="https://github.com/pjmeisch-cc/kotlin-kafka-dsl">Github</a>.</p><h2>The target DSL</h2><p>The following code pieces show the DSL in action. The first example shows a producer application that writes strings into Kafka. The second example is a consumer application that reads them from Kafka.</p><p>I first show the final code that results in using the DSL before explaining how to get there:</p><h3>The producer</h3>

```
fun main(args: Array) {
    val log: Logger = LoggerFactory.getLogger("de.codecentric.ProducerRunnerDSL")
    
    kafka("localhost:9092") {
        producer("kt-topic") {
            (1..10).forEach {
                val msg = "test message $it ${LocalDateTime.now()}"
                log.info("sending $msg")
                send(msg)
            }
            flush()
        }
    }
}

```

<p>In this code I first use the <code>kafka("localhost:9092")</code> function to define the connection to the Kafka broker that is running on the local machine. Within this context I then create a producer for the topic with <code>producer("kt-topic")</code>.</p><p>In the context of the producer there is a loop that produces 10 messages. These messages are sent to Kafka by calling the producer's <code>send(msg)</code> method. After the loop, the <code>flush()</code> method of the producer is called to make sure that the messages are delivered to Kafka before the program terminates.</p><h3>The consumer</h3>

```
fun main(args: Array) {
  val log: Logger = LoggerFactory.getLogger("de.codecentric.ConsumerRunnerDSL")

  kafka("localhost:9092") {
    consumer("kt-topic") {
      Runtime.getRuntime().addShutdownHook(Thread(Runnable {
        stop()
      }))
      consume {
        log.info("got $it")
      }
    }
  }
}

```

<p>Like the producer, the consumer application first defines the connection the Kafka broker. It then creates a consumer for the topic with <code>consumer("kt-topic")</code>. In this consumer, a shutdownhook is added which will call the consumer's <code>stop()</code> method to ensure that the underlying <code>KafkaConsumer</code> is shut down properly on program termination.</p><p>Then the consumer's <code>consume</code> method is called with a lambda function that processes the String which is retrieved from the Kafka topic by logging its value. This <code>consume</code> method blocks on reading the data from Kafka and will only return on an error or when the <code>stop()</code> method has been called from a different thread, for example on program shutdown.</p><h2>Prerequesites</h2><h3>Apache Kafka</h3><p>To get the programs running, it is necessary to have a running <a href="https://kafka.apache.org/">Apache Kafka</a> . I used the <a href="https://docs.confluent.io/current/installation/installing_cp.html">Confluent Platform OpenSource</a> to test my implementation, any other Apache Kafka setup is fine as well, as long as you have the address of a Kafka broker.</p><h3>Kotlin concepts</h3><p>There are two Kotlin language features that are essential for the writing of a DSL like I show it in this article:</p><p><strong>lambda arguments to functions:</strong><br /> When a function <code>func</code> has another function as last argument and in the call to this function this argument is passed in as a lambda, then the argument can be written <strong>after</strong> the function argument call (<a href="https://kotlinlang.org/docs/reference/lambdas.html#passing-a-lambda-to-the-last-parameter">documentation</a> ):</p>

```
// func gets a String as first and a function from String to String as second argument.
// it is calling the second argument with the first as parameter and prints the result
fun func(arg1: String, calledWithArg1: (String) -> String) {
  println(calledWithArg1(arg1))
}

// this might be called like this (don't do it this way!):
func("foo", { s: String -> s + s })
// prints foofoo

// the Kotlin way, moved out of the parenthesis and using the its default parameter:
func("foo") {
  it + it
}
// prints foofoo as well


```

<p><strong>extension functions:</strong><br /> Extension functions offer the possibility to add functionality to existing classes. They are not implemented in the class itself (<a href="https://kotlinlang.org/docs/reference/extensions.html#extension-functions">Documentation)</a>  – – the example is a bit off-topic here</p>

```
class PaymentInfo(var amount: Double, var creditCard: String)

val p = PaymentInfo(42.0, "1234-5678-0123-1234")

// extension function to double the price, not defined within the class
fun PaymentInfo.twice() {
  // here 'this' is the PaymentInfo object
  amount *= 2
}

// call the extension function for an instance of class PaymentInfo
p.twice()
// now the amount is 84.0

// a function to modify a PaymentInfo with an extension passed in as a lambda
fun modifyPayment(payment: PaymentInfo, modifier: PaymentInfo.() -> Unit) {
  payment.modifier()
}

// mask the credit card
modifyPayment(p) {
  // hier ist 'this' das PaymentInfo Objekt
  creditCard = "XXX"
}
// now the creditcard value in the object is masked

```

<p>We can use extension functions to either add functionality to some classes which cannot be modified (like for example <code>String</code> or <code>List</code>).<br /> The other main purpose is that by using an extension function, a lambda can be passed to a function which is then executed in the context of the object – like in the example when the credit card number is masked. The <code>modifyPayment()</code> call in the example again shows the Kotlin way of moving the lambda out of the parenthesis.</p><h2>Step 1: internal classes used to access Apache Kafka</h2><p>In a first step, we create some Kotlin classes to write to and read from Kafka. These classes use the official Kafka client library written in Java. The classes make up a small Kotlin layer around the Kafka client library.</p><h3>Kafka configuration class</h3>

```
data class Kafka(val bootstrapServers: String)

```

<p>This first very simple class just stores the configuration for the connection to the Kafka broker. I kept the configurable properties to the absolute minimum to keep the example small.</p><h3>The Producer clas</h3>

```
class Producer(kafka: Kafka, private val topic: String) {
  private val kafkaProducer: KafkaProducer<String, String>

  init {
    val config = Properties()
    config[ProducerConfig.BOOTSTRAP_SERVERS_CONFIG] = kafka.bootstrapServers
    config[ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java
    config[ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java
    kafkaProducer = KafkaProducer(config)
  }

  fun send(msg: String) {
    kafkaProducer.send(ProducerRecord(topic, msg))
  }

  fun flush() = kafkaProducer.flush()
}

```

<p>The Producer class is initialized with a <code>Kafka</code> configuration object and the topic name. In the <code>init</code> block, a <code>KafkaProducer</code> from the official client library is instantiated with the necessary values.
The <code>send(msg: String)</code> method creates a <code>ProducerRecord</code> and sends it to Kafka, and the <code>flush()</code> is just passed on to the client library.</p><h3>The Consumer class</h3>

```
class Consumer(kafka: Kafka, topic: String) {
  private val kafkaConsumer: KafkaConsumer<String, String>

  @Volatile
  var keepGoing = true

  init {
    val config = Properties()
    config[ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG] = kafka.bootstrapServers
    config[ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
    config[ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
    config[ConsumerConfig.GROUP_ID_CONFIG] = UUID.randomUUID().toString()
    kafkaConsumer = KafkaConsumer<String, String>(config).apply {
      subscribe(listOf(topic))
    }
  }

  fun consume(handler: (value: String) -> Unit) = Thread(Runnable {
    keepGoing = true
    kafkaConsumer.use { kc ->
      while (keepGoing) {
        kc.poll(500)?.forEach {
          handler(it?.value() ?: "???")
        }
      }
    }
  }).start()

  fun stop() {
  keepGoing = false
  }
}

```

<p>This class as well is initialized with the Kafka configuration and topic and creates a <code>KafkaConsumer</code> instance that subscribes to the requested topic. The class has a property named <code>keepGoing</code> which is set to true when its <code>consume(handler: (value: String) -&gt; Unit)</code> method is called. This method starts a new thread that polls the Kafka topic and calls the handler method for the retrieved values.</p><p>The <code>stop()</code> method is used to set the <code>keepRunning</code> flag to false which stops the polling loop and ends the thread.</p><h3>Applications using these classes</h3><p>Here are two applications that use these classes to produce and consume data without using the custom DSL; this is just plain Kotlin code:</p>

```
fun main(args: Array) {
  val log: Logger = LoggerFactory.getLogger("de.codecentric.ProducerRunner")

  val kafka = Kafka("localhost:9092")
  val topic = "kt-topic"

  val producer = Producer(kafka, topic)
  (1..10).forEach {
    val msg = "test message $it ${LocalDateTime.now()}"
    log.info("sending $msg")
    producer.send(msg)
  }
  producer.flush()
}

```

<p> </p>

```
fun main(args: Array) {
  val log: Logger = LoggerFactory.getLogger("de.codecentric.ConsumerRunner")

  val kafka = Kafka("localhost:9092")
  val topic = "kt-topic"

  val consumer = Consumer(kafka, topic)
  Runtime.getRuntime().addShutdownHook(Thread(Runnable {
    consumer.stop()
  }))
  consumer.consume {
    log.info("got $it")
  }
}

```

<h2>Step 2: The code defining the DSL</h2><p>In order to create the DSL using these classes the following code is required:</p>

```
class KafkaDSL(bootstrapServers: String) {
  private val kafka = Kafka(bootstrapServers)

  fun producer(topic: String, doProduce: Producer.() -> Unit) =
    Producer(kafka, topic).doProduce()

  fun consumer(topic: String, doConsume: Consumer.() -> Unit) =
    Consumer(kafka, topic).doConsume()
}

fun kafka(bootstrapServers: String, init: KafkaDSL.() -> Unit) =
  KafkaDSL(bootstrapServers).init()

```

<p>This is all that is necessary to create our DSL. I will explain these classes and methods step by step for both our producer and consumer example.</p><h3>The Producer</h3>

```
kafka("localhost:9092") {
  // this is the body of the KafkaDSL.init() function.
  // In this block, 'this' is the KafkaDSL object.
}

```

<p>This call to the global <code>kafka</code> function creates a <code>KafkaDSL</code> object and passes the bootstrap servers in the constructor. The <code>KafkaDSL</code> object uses this information to create a <code>Kafka</code> configuration object. The second argument – <code>init</code> – to the <code>kafka()</code> function is a Kotlin extension function for the <code>KafkaDSL</code> class which – in Kotlin style – is written after the <code>kafka()</code> parenthesis. This function is called for the newly created <code>KafkaDSL</code> object after creation.</p>

```
kafka("localhost:9092") {
  // this is the body of the KafkaDSL.init() function.
  // In this block, 'this' is the KafkaDSL object.
  producer("kt-topic") {
    // this is the body of the Producer.doProduce() function.
    // In this block, 'this' is the Producer object
  }
}

```

<p>The <code>KafkaDSL.init()</code> function, which has been passed to the <code>KafkaDSL</code> object, calls the method <code>producer("kt-topic"){}</code>. This is a call to the corresponding method of the <code>KafkaDSL</code> class, the first parameter being the topic name and the second being an extension function for the <code>Producer</code> class. This call creates a Producer object with the necessary arguments and then calls the extension function on this Producer object.</p>

```
kafka("localhost:9092") {
  // this is the body of the KafkaDSL.init() function.
  // In this block, 'this' is the KafkaDSL object.
  producer("kt-topic") {
    // this is the body of the Producer.doProduce() function.
    // In this block, 'this' is the Producer object
    (1..10).forEach {
      send("test message $it ${LocalDateTime.now()}") // <- Producer.send()
    }
    flush() // <- Producer.flush()
  }
}

```

<p>In the body of the <code>Producer.doProduce()</code> extension function, the loop creating the messages is implemented and the messages are sent with the <code>send()</code> and <code>flush()</code> methods of the <code>Producer</code> object.</p><h3>The consumer</h3><p>The code for the consumer is similar to the producer code, so I will just explain the differences. First a <code>KafkaDSL</code> object is created. The difference lies in the implementation of the <code>KafkaDSL.init()</code> function:</p>

```
kafka("localhost:9092") {
  // this is the body of the KafkaDSL.init() function.
  // In this block, 'this' is the KafkaDSL object.
  consumer("kt-topic") {
    // this is the body of the Consumer.doConsume() function.
    // In this block, 'this' is the Consumer object
  }
}

```

<p>Here the <code>consumer("kt-topic")</code> method is called which creates a <code>Consumer</code> object and then calls the <code>Consumer.doConsume()</code> extension function which is passed in as second argument.</p>

```
kafka("localhost:9092") {
  // this is the body of the KafkaDSL.init() function.
  // In this block, 'this' is the KafkaDSL object.
  consumer("kt-topic") {
    // this is the body of the Consumer.doConsume() function.
    // In this block, 'this' is the Consumer object
    Runtime.getRuntime().addShutdownHook(Thread(Runnable {
      stop() // <- Consumer.stop()
    }))
    consume { // <- Consumer.consume(handler)
      log.info("got $it")
    }
  }
}

```

<p>The body of the <code>Consumer.doConsume()</code> function first installs the shutdownhook and then calls the <code>Consumer</code>'s <code>consume()</code> method passing in a lambda as handler to process the values read from Kafka.</p><p>And this is all that we need to create our small custom DSL.</p><h2>Conclusion</h2><p>First we wrote some helper classes in Kotlin to have the desired functionality to write to and read from Apache Kafka. And with only one global method and a helper class with two methods, we implemented our DSL for that. This example shows the potential that lies in Kotlin's language constructs like extension functions or lambda arguments to functions. Even more possibilities are in using operator overloading or infix functions. I refrained from using them here to keep the example small.<br /> Coming from Java, these language features seem strange in the first place, but it's definitely worth working through this stuff.</p><p>Hope you enjoyed this excursion into the world of Kotlin.</p>

How to write a Kotlin DSL – e.g. for Apache Kafka

<p>In this blog post, I want to show how some of Kotlin's feature make it easier to write concise code. First, these are functions from the Kotlin standard library. Second, these are custom written functions that use Kotlin language features, extension functions, or lambda arguments.</p><p>I won't go into details of what extensions functions are or focus on syntax details. For this, I refer to the <a href="http://kotlinlang.org/docs/reference" target="_blank">excellent Kotlin documentation</a> .</p><h2>Functions from the Kotlin library</h2><p>In this first part of this blog post, I'd like to introduce some functions and functionality that the standard Kotlin library offers. This is just a small selection which I found useful and I strongly encourage you to check the source code and <a href="http://kotlinlang.org/api/latest/jvm/stdlib/index.html" target="_blank">documentation</a>  by yourself.</p><h3>Argument checking</h3><p>When writing code that is not using any validation framework, you sometimes have to check the validity of function arguments. You can do it in the classical way with:</p>

```
fun someFunction(answer: Int) {
  if(answer != 42) {
    throw IllegalArgumentException("the answer must be 42!")
  }
  // function code
}
```

<p>But the better way is to use <code>require</code> from the standard library, which does exactly the same:</p>

```
fun someFunction(answer: Int) {
  require(answer == 42) {" the answer must be 42!"}
  // function code
}
```

<h3>Returning multiple values from a function</h3><p>Sometimes you need to return two values from a function. One way to do this is by creating a new class to hold this value. An easier way is to use Kotlin's <code>Pair</code> class, the <code>to</code> infix function and a destructuring variable declaration:</p>

```
fun someFunction() : Pair<Int, String> {
  return 42 to "the answer"
}

fun main(args: Array<String>) {
  val (i: Int, s: String) = someFunction()
  log.info("$s is $i") // logs "the answer is 42"
}
```

<h3Time measuring</h3><p>You probably often use some code to measure the execution time of some code like this:</p>

```

val start = System.currentTimeMillis()
// do something    
val end = System.currentTimeMillis()   

log.info("duration: ${end - start} msecs")

```

<p>Kotlin offers two inline functions, <code>measureTimeMillis</code> and <code>measureNanoTime</code>, which can be used in the following way:</p>

```

val duration = measureTimeMillis {
  // do something
}

log.info("duration: $duration msecs")
```

<h3>File IO</h3><p>Have you ever tried to delete a directory in your program that is not empty? There is no function in the JVM classes that will allow you to do this without implementing a recursive descent into the directory and delete contained files and directories first.<br />
Kotlin has an extension function for that:</p>

```
File("/path/to/some/directory").deleteRecursively()
```

<p>If you have the name of a file and just need it without the extension:</p>

```

val f = File("image.jpg")
val name = f.nameWithoutExtension // sets name to "image"

```

<p>Read a file line by line and call a lambda to process each line:</p>

```

File("filename").useLines { line ->
  println(line)
}

```

<p>There are many more utility functions in the <code>kotlin.io</code> package, please check the documentation.</p><h3>Concurrent programming</h3><h4>Reentrant read write locks</h4><p>Did you ever need to have a resource that can be accessed for writing by one thread, but should be able to be read by multiple threads? Java has the <code>ReentrantReadWriteLock</code> class for this, but using it is not easy as you have to take care of interruptions or exceptions when aquiring and releasing the locks.</p><p>Kotlin has you covered with some helper functions (extension functions on the <code>ReentrantReadWriteLock</code> class) where you just pass in the function you want to be executed within the lock:</p>

```
val sharedResource = mutableMapOf<String, String>()
val lock = ReentrantReadWriteLock()

fun readSomeData(key: String) {
  return lock.read {
    sharedResource[key]
  }
}

fun writeSomeData(key: String, value: String) {
  lock.write {
    sharedResource[key] = value
  }
}
```

<h4>Threads and ThreadLocals</h4><p>When using a ThreadLocal, the normal code to initialize a ThreadLocal object looks something like this:</p>

```
class ClassWithThreadState {
  private val state = object : ThreadLocal<String>() {
    override fun initialValue(): String = "initialValue"
  }

  fun someFunction() {
    var currentState = state.get()
    log.info { "currentState: $currentState" }
    state.set("newState")
  }
}
```

<p>Kotlin adds a <code>getOrSet</code> extension function which makes it easier to set up the inital value:</p>

```
class ClassWithThreadState {
  private val state = ThreadLocal<String>()

  fun someFunction() {
    val currentState = state.getOrSet { "initalValue" }
    log.info { "currentState: $currentState" }
    state.set("newState")
  }
}
```

<p>To run some code in a thread, you have to wrap it in a <code>Runnable</code>, which is then passed into a  <code>Thread</code>, which is then started:</p>

```
fun main(args: Array<String>) {
  val classWithThreadState = ClassWithThreadState()
  Thread(Runnable { classWithThreadState.someFunction() }).start()
}
```

<p>The Kotlin library has a <code>thread</code> function that makes it easier:</p>

```
fun main(args: Array<String>) {
  val classWithThreadState = ClassWithThreadState()
  // imediately start the thread with the runnable
  thread { classWithThreadState.someFunction() }
    
  // or first create it and start it later:
  val t = thread(start = false) { classWithThreadState.someFunction() }
  t.start()
}
```

<p>Apart from the <code>start</code> parameter, the <code>thread</code> function has other parameters which allow you to set the priority, the name or the daemon state of the thread.</p><h2>Writing custom functions and extensions</h2><p>In the second part of this blog, I now want to show how you can easily write functions to make the code cleaner and more precise. These functions also enable you to remove boilerplate code.</p><p>The principles I use for these functions are the same that are used throughout the Kotlin library: extension functions and functions taking a lambda as last argument, which is then written after the parenthesis. For more information about these techniques, you might want to read my blog post <a href="/wissens-hub/blog/kotlin-dsl-apache-kafka">“ Wie schreibt man eine Kotlin-DSL“</a> .</p><h3>Slf4j</h3><p>Are you using <em>slf4j</em> and are tired of writing the same boilerplate code for creating a logger in every class like this?</p>

```
class SomeClass {
  fun doSomething() {
    log.info("doing something")
  }
    
  companion object {
    // always have to repeat the name of the class here:
    private val log: Logger = LoggerFactory.getLogger(SomeClass::class.java)
  }
}
```

<p>With a little extension function on <code>Any</code>, you can easily create an <em>slf4j</em> logger for any class, even if you put the logger in the companion object:</p>

```
fun Any.logger(): Logger {
  val clazz = if (this::class.isCompanion) this::class.java.enclosingClass else this::class.java
  return LoggerFactory.getLogger(clazz)
}
```

<p>When using this function, you just can skip passing in the class into the call to create the logger:</p>

```

package de.codecentric.kotlingoodies
import logger

class SomeClass {
  fun doSomething() {
    log.info("doing something")
  }

  companion object {
    // creates a logger with the name "de.codecentric.kotlingoodies.SomeClass"
    private val log = logger()
  }
}
```

<p>Are you checking logging levels of a logger to prevent, for example, string templates from being evaluated with code like this?</p>

```
fun someFunction() {
  val someObject = Any() // or something else, doesn't matter here

  try {
    someObject.doSomething()
    if(log.isDebugEnabled) {
      log.debug("doing something with $someObject")
    }
  } catch (e: Exception) {
    if(log.isWarnEnabled) {
      log.warn("could not do something with $someObject" e)
    }
  }
}
```

<p>With just a handful of extension functions for the <code>Logger</code> class you can reduce your code (just showing for debug and info level):</p>

```
inline fun Logger.info(msg: (() -> String)) {if (isInfoEnabled) info(msg())}
inline fun Logger.info(t: Throwable, msg: (() -> String)) {if (isInfoEnabled) info(msg(), t)}
inline fun Logger.debug(msg: (() -> String)) {if (isDebugEnabled) debug(msg())}
inline fun Logger.debug(t: Throwable, msg: (() -> String)) {if (isDebugEnabled) debug(msg(), t)}

// use it like this: 
fun someFunction() {
  val someObject = Any() // or something else, doesn't matter here

  try {
    someObject.doSomething()
    log.debug { "doing something with $someObject" }
  } catch (e: Exception) {
    log.warn(e) { "could not do something with $someObject" }
  }
}
```

<p>This makes your code more readable and prevents you from the trap of changing the logged level for example from <em>info</em> to <em>debug</em> and forgetting to change from <em>isInfoEnabled</em> to <em>isDebugEnabled</em> as well.</p><h3>Writing functions to execute boilerplate code around something</h3><p>Imagine that in your code you have lots of places where you for example start a transaction, do something, commit the transaction on success or roll back the transaction in an error case.</p><p>To handle this more elegantly, you can define a function – it has to be inline and reified to keep the result type. In this example it is called <code>transactional</code>, that does the boilerplate part and calls the code that is to be executed (<code>R</code> is the return type of the code to be executed):</p>

```
inline fun <reified R> transactional(f: () -> R): R {
    // code to begin transaction
  return try {
    f()
    // code to commit transaction
  } catch(e: Exception){
    // code to rollback transaction
    throw e
  }
}
```

<p>Then, to use it, you just pass a lambda to this function:</p>

```
data class Record(val name: String)
fun main(args: Array<String>) {

  val records = transactional { 
    // this code is run within a transaction and is probably more complex in a real world scenario
    listOf(Record("John"), Record("James"))
  }

  records.forEach { println(it) }
}
```

<h3>Removing diacritics from strings</h3><p>Just recently, I needed a function to remove diacritics (for example accents) from words because we were building URL paths with this words. This can be done by writing a function with the following code:</p>

```
fun noDiacritics (s: String) : String {
  val normalized = Normalizer.normalize(this, Normalizer.Form.NFD)
  val stripped = normalized.replace("\\p{M}".toRegex(), "")
  return stripped
}
```

<pThe better variant is to write this as an extension function for the String class which makes it possible to use it on String variables and even on String constants:</p>

```

inline fun String?.noDiacritics() = this?.let{Normalizer.normalize(this, Normalizer.Form.NFD)
  .replace("\\p{M}".toRegex(), "")}

fun main(args: Array<String>) {
  println("Porsche Coupé".noDiacritics())
  // output is: Porsche Coupe
}
```

<h2>Summary</h2><p>In this post, I showed how small functions – either from the Kotlin standard library or self-written functions – can reduce the code you have to write. The examples also showed how they can make your code easier to read, understand, and maintain.</p><p>Kotlin features such as extension functions, inline, and infix functions help you create such helper functions yourself.</p><p>For further reference, I can only recommend the source code of the different Kotlin library functions, they are excellent material to study and learn from.</p>

Kotlin goodies that make a developer's life easier

 Senior IT Consultant 

<h2><strong>Was ist die Hacker School?</strong></h2><p>Die Hacker School ist ein gemeinnütziger Verein aus Hamburg, der sich zum Ziel gemacht hat, Kinder und Jugendliche IT-technisch zu fördern und sie für’s Programmieren zu begeistern.</p><p>Am Wochenende vom 13. und 14. Juli haben 17 Unternehmen aus Karlsruhe, unter anderem codecentric, eine Hacker School veranstaltet, um Schülern über die zwei Tage das Interesse an IT zu wecken.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/z0lnRzRxiwIMcCaGYbwB"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/z0lnRzRxiwIMcCaGYbwB" alt width="400" height="400" /></a> </p><p>Betreut von zwei Organisatoren der Hacker School und vier Mitarbeitern der codecentric konnten 20 Schüler im Alter von 11-16 Jahren zwischen zwei Projekten frei wählen. Das Projekt, welches von den Kollegen der Hacker School betreut wurde, wurde mit Hilfe des Einplatinencomputer <a href="https://microbit.org/de/">microbit</a>  realisiert. Unser Projekt hatte zum Ziel, mithilfe von <a href="https://www.python.org/">Python </a> und einem Raspberry Pi 4 im Spiel <a href="https://www.minecraft.net/de-de/">Minecraft</a>  ein spielbares Klavier zu programmieren – aber davon im nächsten Abschnitt mehr. Die Kinder und Jugendlichen arbeiteten jeweils in Zweierteams an ihrem Projekt.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/0dJ4VKhESOutwYM1mVqw"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/0dJ4VKhESOutwYM1mVqw" alt="Microbit Setup" width="700" height="525" /></a> </p><p>Die Zusammenarbeit zu zweit fördert den Teamgedanken, so konnten sich die Schüler gegenseitig unterstützen, bevor sie auf Hilfe von einem Erwachsenen angewiesen waren. Auch in der Arbeitswelt wird diese Form von Zusammenarbeit mit der beliebten Technik “Pair Programming” – ein Rechner, zwei Entwickler – praktiziert. Dabei wird zwischen den Rollen Fahrer und Navigator unterschieden. Der Fahrer schreibt den Code und der Navigator überprüft diesen zeitgleich. Die Rollen werden dabei in regelmäßigen Abständen möglichst häufig getauscht.</p><p>Beide Projekte fanden guten Anklang, das Interesse der Jugendlichen war geweckt.</p><p> </p><h2><strong>Unser Minecraft + SonicPi + Python Projekt</strong></h2><p>Um den Schülern in spielerischer Weise das Programmieren beizubringen und um sicherzustellen, dass sie frühe Erfolge erleben, entschieden wir uns für ein mehrstufiges Projekt auf der Basis des Spiels Minecraft. <a href="https://www.minecraft.net/de-de/">Minecraft</a>  ist ein Open-World-Spiel, in der die Spieler in einer <a href="https://de.wikipedia.org/wiki/3D-Computergrafik">3D</a> -Welt aus würfelförmigen Blöcken Konstruktionen und Gebäude bauen können. Die Basis des Projekts ist auf der offiziellen Seite für <a href="https://projects.raspberrypi.org/en/projects/the-big-minecraft-piano">Beispielprojekte mit dem Raspberry Pi</a>  zu finden.</p><p>Unser gemeinsames Ziel war es, ein voll funktionsfähiges Klavier programmatisch mit Hilfe von Python 3 zu bauen. Das Klavier sollte durch Hüpfen der Spielfigur auf den jeweiligen Tasten verschiedene Töne wiedergeben. Realisiert wurde dies durch <a href="http://sonic-pi.net/">Sonic Pi</a> , ein Musik-Synthesizer auf dem Raspberry Pi, mit dem Musik programmiert werden kann.</p><p>Weshalb haben wir uns für dieses vergleichsweise anspruchsvolle Projekt entschieden? Wir hatten vermutet, dass ein Python-Programm ohne die Anbindung zu anderen Programmen, Schülern zum Einen weniger Spaß und zum Anderen womöglich zu wenig Herausforderungen bieten würde. Aufgrund der Kombination von Minecraft, Python und Sonic Pi, war sowohl die Aufgabenmenge als auch die kreative Freiheit, ideal.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/f8zJ2FMEQRy1gDAg0cQw"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/f8zJ2FMEQRy1gDAg0cQw" alt="Konzentriertes Arbeiten" width="700" height="582" /></a> </p><p> </p><h2><strong>Kursstruktur</strong></h2><p>Zu Beginn packten die Schüler die Raspberry PI’s mit allem Zubehör aus und bauten sie zusammen.</p><p>Unser Kurs war in folgende Schritte aufgeteilt:</p><ul><li>Einführung in Python (Schleifen / Bedingungen / Ausgabe / Methoden)</li><li>Einführung in das Spiel Minecraft (einige Schüler wussten mehr als wir 🙂 )</li><li>Einführung in den Musik-Synthesizer Sonic Pi</li><li>Bau eines Klaviers in Minecraft (manuell)</li><li>Einfache Manipulation der Minecraft-Welt mit Python (Erste Erfahrungen mit der Minecraft API in Python sammeln)</li><li>Klavier programmatisch erzeugen lassen</li><li>Klavier mit Python verbinden (Wie erkenne ich, welche Taste gedrückt wurde? Wo befindet sich der Spieler? Welche Elemente befinden sich ober/unterhalb des Spielers?)</li><li>Python mit Sonic Pi verbinden</li></ul><p> </p><h2><strong>Herausforderungen</strong></h2><p>Zu Beginn wurden wir von Hardware-Problemen geplagt – so startete bei der Hälfte der Teams Minecraft im Vollbildmodus und ließ sich nicht mehr schließen, was dazu geführt hat, dass Schüler eine Zeit lang nicht mit Minecraft programmieren konnten. Es stellte sich als Kompatibilitäts-Problem mit den verwendeten Monitoren heraus – das Wechseln des Monitors schaffte Abhilfe.</p><p>Darüber hinaus hatten wir Schwierigkeiten mit der vorinstallierten Version von Sonic Pi. Diese war auf dem neuen Raspberry Pi 4 in einer zu alten Version installiert und ein Update war nicht ohne weiteres möglich (<a href="https://github.com/samaaron/sonic-pi/issues/2095">Github Issue</a> ). Wir bemerkten dies rechtzeitig und konnten das Problem so am zweiten Tag umgehen. Auf dem Raspberry Pi 3, auf dem wir zuvor den Kurs testeten, trat dieses Problem nicht auf. Diese Lektion war nichtsdestotrotz lehrreich, da die Schüler so lernen konnten, weshalb man Software immer auf der endgültigen Hardware testen sollte ;-).</p><p>Zudem lernten wir, dass die Schnittstelle von Python zu Minecraft, als auch zu Sonic Pi mit Wechselwirkungen einhergeht und somit deutlich komplizierter zu benutzen ist als ursprünglich geplant. Jüngere Schüler brauchten an dieser Stelle die Unterstützung von uns Betreuern. Obwohl wir an diesem Wochenende einige neue und nicht-triviale Konzepte eingeführt haben (3D Koordinatensystem, Python, Schnittstellen, Variablen/Schleifen/Datentypen etc.), haben sich die Schüler tapfer geschlagen und die Ergebnisse können sich sehen lassen.</p><p> </p><h2><strong>Fazit</strong></h2><p>Für uns war die Hacker School eine spannende Erfahrung, in der wir unser Know-How an die nächste Generation weitergeben konnten und dabei viel Spaß hatten. Die Schüler waren ebenfalls zufrieden, jedes Team konnte zum Schluss ein selbst programmiertes funktionales Minecraft-Klavier präsentieren.</p><p>Es war auch sehr schön zu sehen, dass die Gruppen bei der Lösungsfindung besonders kreativ waren. So erhielt jedes einzelne Klavier einen eigenen Charme und Spezialfunktionen. Zum Beispiel baute ein Team eine Funktion, der das Lied “Alle meine Entchen” abspielte, wenn man einen bestimmten Minecraft-Block berührt hat.</p><div><dl><dt> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GXshbz9bTU24A3Z3oWkh"><img width="150" height="150" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GXshbz9bTU24A3Z3oWkh" alt="Minecraft Piano 1" loading="lazy" /></a> </dt></dl><dl><dt> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/xjm1r3CtQzWU6FH2MvLn"><img width="150" height="150" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/xjm1r3CtQzWU6FH2MvLn" alt="Minecraft Piano 2" loading="lazy" /></a> </dt></dl><dl><dt> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/68DIXqIQR6m3iBpowRnQ"><img width="150" height="150" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/68DIXqIQR6m3iBpowRnQ" alt="Minecraft Piano 3" loading="lazy" /></a> </dt></dl><br /></div><p> </p><p>Die Kinder und Jugendlichen waren begeistert und berechtigterweise stolz auf ihre Leistung. Den Quellcode und die gedruckte Dokumentation konnten sie zum Schluss mit nach Hause nehmen.</p><p>Uns hat es viel Spaß gemacht und wir freuen uns auf nächstes Jahr! Frie Preu, Paul Strobel, Achim Nierbeck, Felix Magnus und Manuel Wessner</p><p> </p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/vVOYxi9SBmhIl002JLIy"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/vVOYxi9SBmhIl002JLIy" alt="Zwei Schüler, zufrieden mit ihrer Arbeit" width="700" height="525" /></a> </p><p>PS: Da das Angebot so gut ankam, fragten viele Eltern noch nach weiteren, ähnlichen Angeboten. Die Initiative devvox4kids plant <a href="http://www.devoxx4kids.de/karlsruhe/">aktuell eine Veranstaltung im Oktober</a>  in Karlsruhe.</p><p> </p><h3><strong>Links:</strong></h3><ul><li><a href="https://www.hacker-school.de/">https://www.hacker-school.de/</a> </li><li><a href="http://www.devoxx4kids.de/">http://www.devoxx4kids.de/</a> </li></ul>

Hacker School in Karlsruhe: Auf die Plätze, fertig, hacken!

<p>Bei der Entwicklung von Software kann es vorkommen, dass die Weiterarbeit an einem Feature durch projektinterne Abhängigkeiten aufgehalten wird. Ein Beispiel hierfür ist die getrennte Entwicklung von Frontend und Backend. Oft kann gewisse Funktionalität erst im Frontend umgesetzt werden, wenn das Backend dafür über eine entsprechende Schnittstelle Daten anbietet.</p><p>Um diese Abhängigkeit aufzulösen, gibt es mehrere Ansätze. Ein solcher Ansatz ist der Einsatz von Mock-Servern, den ich in diesem Blogpost vorstelle möchte.</p><h2>Was sind eigentlich Mock-Server?</h2><p>Mock-Server können genutzt werden um die Funktionalität eines “echten” Backends vorzutäuschen und stellen damit einen leichtgewichtigen Ansprechpartner für ein Frontend da.</p><p>Um einen Mock-Server zu betreiben, muss lediglich die zu entwickelnde API definiert sein. Diese Definition kann entweder informell oder über eine API Spezifikation á la OpenAPI Specification erfolgen. Dann kann der Mock-Server entsprechend konfiguriert werden sodass er über verschiedene Endpunkte gemockte Antworten zurückliefert.</p><p>Mit minimalem zusätzlichen Konfigurations- / Entwicklungsaufwand kann man also anfangen, ein mit einem Service kompatibles Frontend zu entwickeln, bevor der Service selbst fertig implementiert ist, ganz ohne Verzögerungen oder Abhängigkeiten.</p><p>Neben der beschriebenen Entkopplung von Frontend- und Backend-Entwicklung, bringt der Einsatz eines Mock-Servers weitere Vorteile:</p><ul><li>Einen Mock-Server zu starten ist oft deutlich leichtgewichtiger, als ein vollständiges Backend zu starten, man spart also Ressourcen bei der Entwicklung des Frontends</li><li>Ein Mock-Server läuft üblicherweise lokal, was den Vorteil hat, dass auch offline weiterentwickelt werden kann</li><li>Mock-Server können für UI-Tests genutzt werden um verschiedene Fehlerfälle zu simulieren und zu testen, die im Kontext eines echten Systems unter Umständen schwer reproduzierbar sind</li><li>Auch für Prototyping eignen sich Mock-Server sehr gut, man kann schnell Ideen für das Frontend einer Applikation visualisieren und muss sich nicht um die Implementierung zugehöriger Services im Backend kümmern</li></ul><h3>Implementierung eines Mockservers mit json-server</h3><p>Eine Möglichkeit, einen Mock-Server aufzusetzen, ist der Einsatz von <a href="https://nodejs.org/en/" target="_blank">Node.js</a>  zusammen mit dem Package <a href="https://github.com/typicode/json-server" target="_blank">json-server</a> .</p><p>Das Package <strong>json-server</strong> erlaubt es dem Benutzer, die API-Antworten des Mock-Servers als JSON-Datei zu definieren. Ein einfaches Beispiel hierfür ist das folgende:</p><p>Man installiert sich global das json-server-Package mit</p><p><code>npm install -g json-server</code></p><p>und erstellt sich eine <code>db.json</code> Datei mit folgendem Inhalt:</p><p></p><p>Danach startet man den Mock-Server basierend auf dieser “Datenbank” via</p><p><code>json-server --watch db.json</code>.</p><p>Jetzt sind die in der JSON-Datei hinterlegten Daten über eine REST-Schnittstelle via http://localhost:3000/posts/1 etc. erreichbar und können von einem Frontend abgefragt werden.</p><p>Das Frontend kann außerdem POST, PUT, PATCH und DELETE requests (mit <em>Content-Type: application/json</em>) an den Endpunkt schicken – diese werden dann auf die hinterlegte „Datenbank” angewandt.</p><p>Nichtsdestotrotz reicht diese Funktionalität manchmal nicht aus, um alle gewünschten Verhaltensweisen des Services abzubilden. In diesem Fall kann man json-server auch programmatisch in einem eigenen Node.js-Programm nutzen. Voraussetzung hierfür ist, dass json-server als Dependency vom Projekt installiert wurde (also nicht global).</p><p><code>server.js</code></p><p></p><p>Der „selbstgebaute“ Mock-Server kann dann via <code>node server.js</code> gestartet werden.</p><p>json-server erlaubt es hierbei, weitere <a href="https://expressjs.com/de/" target="_blank">express</a>  Middlewares (z.B. für Authentifizierung oder Validierungen) hinzuzufügen oder weitere eigene Verhaltensweisen zu definieren.</p><h3>json-server + Typescript = ❤️</h3><p>Die programmatische Nutzung von json-server ist auch in <a href="/wissens-hub/blog/web-anwendungen-mit-node-express-typescript-entwickeln">TypeScript</a> -Projekten möglich, hierzu muss einfach die Typdefinition für json-server über das Package <em>@types/json-server</em> installiert werden.</p><p>Der Einsatz von TypeScript bietet hier den weiteren Vorteil, dass die Datenbank auch als TypeScript-Datei angelegt werden kann, was eine Typisierung der zurückgegebenen Mock-Daten ermöglicht. Das erleichtert vor allem in komplexen Anwendungsfällen die Strukturierung der Daten und ist insbesondere dann praktisch, wenn man ohnehin ein Tool wie <a href="https://github.com/OpenAPITools/openapi-generator" target="_blank">openapi-generator</a>  nutzt, um sich basierend auf der API Spezifikation TypeScript-Typdefinitionen generieren lässt.</p><p>Hier ein Beispiel, welches json-server in Kombination mit TypeScript nutzt:</p><p><code>package.json</code></p><p></p><p>Zuerst definieren wir die <code>package.json</code> Datei. Diese enthält alle benötigten Dependencies. Wir installieren die Dependencies via <code>npm install</code>.</p><p><code>db.ts</code></p><p></p><p>Als nächstes setzen wir unsere „Datenbank“ auf, aber anstelle einer JSON-Datei definieren wir sie diesmal in einer TypeScript-Datei und beschreiben zusätzlich die Datentypen unserer Mockdaten.</p><p><code>mockserver.ts</code></p><p></p><p>Zuletzt erstellen wir analog zum ersten Beispiel eine TypeScript-Datei, die die TypeScript-Datenbank importiert und diese anstelle der zuvor genutzten JSON-Datei an json-server übergibt.</p><p>Ähnlich wie beim ersten Beispiel sind nach dem Start des Mock-Servers via <code>npm start</code> die in der <code>db.ts</code> Datei hinterlegten Daten über eine REST-Schnittstelle via http://localhost:3000/posts/1 etc. erreichbar. Auch hier kann das Frontend wieder Daten über die Standard HTTP-Methoden abfragen und ändern.</p><h2>Fazit</h2><p>Mit Node.js, Typescript und json-server lassen sich mit minimalem Konfigurations- bzw. Entwicklungsaufwand einfach Mock-Server konfigurieren, die Abhängigkeiten auflösen und dadurch das Entwickeln eines Frontends deutlich beschleunigen können. json-server bietet hierbei auch den Vorteil, mit bereits aus express bekannten Methoden einfach erweiterbar zu sein. Der Einsatz von TypeScript im Kombination mit einer API Spezifikation und openapi-generator stellt sicher, dass die gemockten Daten exakt der API-Spezifikation entsprechen.</p><p>Alternativen zu einem mit json-server “selbstgebauten” Mock-Server existieren viele: <a href="https://www.mock-server.com/" target="_blank">MockServer</a> , <a href="http://wiremock.org/" target="_blank">Wiremock</a> , <a href="https://github.com/danielgtaylor/apisprout" target="_blank">Apisprout</a>  und viele mehr bieten ähnliche Funktionalitäten, setzen aber ihren Fokus anders und bilden unterschiedliche Anwendungsfälle ab. Gerade im Frontend-Prototyping-Bereich überzeugt mich json-server aber durch den niedrigen Konfigurationsaufwand.</p>

Schnelle Frontend-Entwicklung durch typisierte Mock-Server mit json-server und TypeScript

<h2>Was ist Deno?</h2><div> <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/fgmBn1umRXSrJX6XYHMB" alt="Deno Logo" width="200" height="200" /></div><p> <a href="https://deno.land/" target="_blank">Deno</a>  (ein Anagramm von „Node“ 🤯), ist eine JavaScript und TypeScript Runtime, die<br /> seit Mai 2020 in der Version 1.0 verügbar ist. Deno wurde von Ryan Dahl, dem ursprünglichen Entwickler von Node.js, entwickelt und soll einige konzeptionelle<br /> Einschränkungen und Mängel von Node.js beheben.</p><p>Deno ist Open-Source (MIT) in Rust programmiert, basiert auf der V8 JavaScript Engine und inkludiert einen<br /> TypeScript Compiler. Dieser erlaubt es neben JavaScript auch direkt TypeScript Programme über die Kommandozeile auszuführen.</p><h2>Was ist das besondere an Deno?</h2><p>Deno soll einige bekannte Mängel von Node.js beheben. Dazu zählen unter anderem:</p><p><strong>Ein schlecht konzipiertes Modulsystem mit einem stark zentralisiertem<br /> Verteilungsmechanismus.</strong></p><ul><li>Deno setzt hier folgendermaßen an:</li><ul><li> Deno verwendet ECMAScript Module (anstelle der bei Node.js üblichen<br /> CommonJS Module)</li><li> Deno Module können überall gehostet werden – ein zentrales Repository (wie<br /> bei Node.js das npm Registry) gibt es nicht. Module können direkt über<br /> ihre URL importiert werden und werden immer lokal gecached und kompiliert.<br /> Außerdem werden Module nicht auf neuere Versionen aktualisiert, außer wenn<br /> man das ausdrücklich angibt.</li></ul></ul><p><strong>Viele veraltete Legacy-APIs, die aber weiterhin unterstützt werden<br /> müssen.</strong></p><ul><li>Das macht Deno anders:</li><ul><li> Module via <code>require()</code> zu importieren wird nicht unterstützt,<br /> außschließlich der ECMAScript Modul <code>import</code> Syntax wird<br /> unterstützt</li><li> Deno verwendet moderne ECMAScript-Funktionen in allem internen APIs und<br /> Standardbibliotheken. Alle asynchronen Funktionen nutzen Promises – eine<br /> auf Callbacks basierende API wie bei Node.js gibt es nicht.</li></ul></ul><p><strong>Fehlende Berücksichtigung von Security-Aspekten.</strong></p><ul><li>Deno verhält sich hier anders, denn:</li><ul><li> Standardmäßig dürfen Programme, die mit Deno ausgeführt werden, nicht auf<br /> die Festplatte, das Netzwerk, auf Unterprozesse oder auf<br /> Umgebungsvariablen zugreifen. Dies muss explizit über<br /> Kommandozeilenargumente erlaubt werden (z.B.: <code>--allow-read</code>,<br /> <code>--allow-net=api.codecentric.com</code>, etc.)</li><li> Bei unbehandelten Fehlern stürzt die Deno Runtime (im Gegensatz zu<br /> Node.js) immer direkt ab. Dadurch werden nicht vorhersehbare Ereignisse<br /> verhindert, die auftreten können, wenn ein Programm nach einem<br /> unbehandelten Fehlern weiterläuft.</li></ul></ul><p>Zudem soll Deno out-of-the-box eine sehr gute Developer-Experience bieten, u.A.<br /> durch:</p><ul><li>TypeScript Support</li><li> Eingebautes Tooling:<ul><li>Unit Testing (<code>deno test</code>)</li><li>Code Formatting (<code>deno fmt</code>)</li><li>Linting (<code>deno lint</code>)</li><li>Dependency Checking (<code>deno info</code>)</li></ul></li><li> Eingebauter live-reload File-Watcher (seit Deno 1.4):<br /> <code>deno run --watch</code>, muss aktuell zusätzlich noch mit der<br /> <code>--unstable</code> Flag ausgeführt werden, da das File-Watcher-Feature<br /> noch nicht stabil ist</li><li> Eingebautes Bundling (via <code>deno bundle</code>) um alle verwendeten<br /> Module und den eigenen Code in eine einzige JavaScript Datei zu bündeln</li></ul><h2>Projekt-Setup & Hello World</h2><p>Doch genug der Theorie, wie startet man am besten mit der Deno Entwicklung?<br /></p><ol><li> Deno installieren: Die Anleitungen dazu findet ihr auf der offiziellen<br /> <a href="https://deno.land/" target="_blank">Homepage von Deno</a> </li><p><li> Gehostetes „Hello-World“ Modul ausführen<ul><li> Das „Hello-World“ Modul ist hier gehostet:<br /> <a href="https://deno.land/std@0.69.0/examples/welcome.ts" target="_blank">https://deno.land/std@0.69.0/examples/welcome.ts</a> </li><p> <em><strong>Tipp: clickt auf den Modul-Link, alle bei Deno-Land gehostete Module<br /> können auch direkt im Browser inspiziert werden!</strong></em></p><li>Programm über die Kommandozeile ausführen:</li>

```
deno run https://deno.land/std@0.69.0/examples/welcome.ts
```

<li>Ergebnis:</li>

```

Download https://deno.land/std@0.69.0/examples/welcome.ts
Check https://deno.land/std@0.69.0/examples/welcome.ts
Welcome to Deno 🦕
```

 </ul> </li> <li>Eigenes "Hello-World" Modul ausführen</li> </p><ul> <li> Legt eine <strong><code>hello.ts</code></strong> Datei an </li> <li>Befüllt sie mit eurem Hello World Code, z.B.</li> 

```
console.log("Hello codecentric 👋");
```

<li>Führt sie mit <code>deno run hello.ts</code> aus</li><li>Ergebnis:</li>

```

Check file:///path/to/file/hello.ts
Hello codecentric 👋
```

 </ul> </ol> <h2>Eine einfache REST API mit Deno & Oak</h2> Um etwas über das Hello-World Beispiel hinaus einen Einblick in den aktuellen Entwicklungsstand von Deno zu bekommen, wollen wir in diesem Abschnitt eine sehr einfache REST API bauen. Hierzu nutzen wir das Framework <a href="https://github.com/oakserver/oak" target="_blank">oak</a> . Oak ist ein Middleware-Framework für das in Deno inkludierte <code>http</code> Modul. Leser\*innen, die bereits mit <strong>Node.js</strong> und <strong>Express</strong> oder <strong>Koa</strong> gearbeitet haben, dürften sich im folgenden Code wie Zuhause fühlen 😉. <ul> <li>Zuerst installieren wir oak als dependency via <code>npm install ..</code></li> <li><strong><em>Ach nein, halt! Da war ja was...</em></strong></li> <li>In Deno werden alle dependencies einfach in der TypeScript Datei, in der sie verwendet werden, direkt über eine URL importiert!</li> <li>Wir legen uns also eine Datei <strong><code>app.ts</code></strong> an, und in diese kommt der folgende Code:</li> </ul> 

Deno – Einführung & Entwicklung einer einfachen REST API

IT Consultant

Test-Fixtures: Wozu denn überhaupt?

<a href="https://graphql.org">GraphQL</a> excels by suggesting to think more from the actual client requirements than from the sometimes prematurely abstract server-side grand conceptualizations. It's essential to strike the right balance between the perspectives of <em>all</em> stakeholders, foremost the clients, to evolve truly superb APIs; and APIs need to be truly superb to remain usable and extensible over time, as complexity inevitably will grow. While the GraphQL Schema allows clients to see what's available, clients have to specify what they need in actuality, i.e. which fields in particular, allowing the server to measure the success of every single field; and eventually remove fields that are not needed any more. This is crucial to know, especially when migrating APIs. This way to think about APIs should also hold true when a service needs to report business errors. Let's explore the options GraphQL gives us.
<h2 id="_technical_errors">Technical Errors</h2>

In GraphQL, a response can contain not only the <code>data</code>, but also a list of <code>errors</code> to report technical problems. For example, when a request is syntactically invalid, the service should be as assistive as possible, so the client can easily understand what they can do to fix it, e.g. by responding:
<div class="listingblock">
<div class="title">Listing 1. Validation Error</div>
<div class="content">
<pre class="highlight"><code class="language-json" data-lang="json">{
  "errors": [
    {
      "message": "The type 'Order' doesn't have a field 'foo'. Please check the Schema.",
      "locations": [{"line": 4, "column": 5}],
      "extensions": {
        "description": "Field 'foo' in type 'Order' is undefined",
        "queryPath": ["order", "foo"]
      }
    }
  ],
  "data": null
}</code></pre>
</div>
</div>

If, on the other hand, there is a problem within a service, it should indicate to the client that either a) a retry is the appropriate reaction to a temporary glitch:

<div class="listingblock">
<div class="title">Listing 2. Availability Error</div>
<div class="content">
<pre class="highlight"><code class="language-json" data-lang="json">{
  "errors": [
    {
      "message": "We have a scheduled downtime. Please retry your request later.",
      "extensions": {
        "retryAfter": "2023-05-11T17:53:13.523Z"
      }
    }
  ],
  "data": null
}</code></pre>
</div>
</div>

Or b), there's obviously a bug in the service, so it's nice to inform the clients if and what they can do to have it fixed.

<div class="listingblock">
<div class="title">Listing 3. Internal Bug</div>
<div class="content">
<pre class="highlight"><code class="language-json" data-lang="json">{
  "errors": [
    {
      "message": "Sorry, we have a bug in retrieving the Customer. If you need to see our progress in fixing it, please follow the link below.",
      "extensions": {
        "queryPath": ["order", "customer"],
        "bugfixUrl": "https://bugs.example.org/error-instance/0305a282-9941-4714-83a8-08e5a3353a12"
      }
    }
  ],
  "data": {
    "order": {
      "id": "2022/183435",
      "orderDate": "2023-05-11T17:53:13.523Z",
      "customer": null
    }
  }
}</code></pre>
</div>
</div>

Note the <code>queryPath</code> extension, which allows us to link an error to a specific field within the requested data. GraphQL allows for partial results, i.e. if only part of the request can't be fulfilled, e.g. <code>customer</code>, the data is filled with what <em>can</em> be retrieved, while the errors explain the missing parts.

These examples are technical errors: automated handling (other than retrying) is not reasonable. They can happen anytime and everywhere, so they don't have to be part of the Schema.

But there's also a different class of reasons for some data to be unavailable. Maybe an order is being blocked and hidden because it was considered to be fraudulent. Let's call such situations business errors. They are not exceptions, as they are part of the normal operations of the system; nobody should worry; everything's fine; no stacktraces in the logs, please.

If we report such errors via the same <code>errors</code> mechanism, the clients will ask us for the list of all possible errors and if we can introduce for example some <code>code</code> extension, so they can properly react to them instead of parsing the human-readable message, which could change at any time. But this would be an out-of-band communication: clients would have to consult a second source beside the Schema, e.g. a wiki; and wikis, as the saying goes, are where information goes to die: they are bound to be outdated. We'd rather have business errors be directly part of the Schema.

<h2 id="_result_union">Result Union</h2>

There is an excellent <a href="https://www.youtube.com/watch?v=RDNTP66oY2o">talk by Sasha Solomon</a>, that inspired me to rethink the whole topic and write this blog post. She shows exactly why we need business errors to be result types made visible in the Schema, and provides an ingenious way to do so: the valid response value and all possible business errors are wrapped into a Union result:

<div class="listingblock">
<div class="title">Listing 4. OrderResult Union</div>
<div class="content">
<pre class="highlight">
<code class="language-graphql" data-lang="graphql">type Query {
    order(id: String): OrderResult
}
union OrderResult = Order | OrderNotFound | OrderLocked
type Order {
    id: ID
    orderDate: Date
}
type OrderNotFound &#9312; {
    positive: Boolean
}
type OrderLocked {
    start: DateTime
    reason: LockReason
}
enum LockReason {
    FRAUD
    MANUAL
}</code></pre>
</div>
</div>

<table>
<tbody>
<tr>
<td><i class="conum" data-value="1"></i>①</td>
<td>The <code>positive</code> field is just a dummy value that is always <code>true</code>. I define it here because I want all errors to be Types and not Scalars, so I can add real fields later, and a Type must have at least one field.</td>
</tr>
</tbody>
</table>

A query could be:
<div class="listingblock">
<div class="title">Listing 5. OrderResult Query</div>
<div class="content">
<pre class="highlight"><code class="language-graphql" data-lang="graphql">query order($id: String) {
    order(id: $id) {
        __typename
        ... on Order {
            id
            orderDate
        }
        ... on OrderLocked {
            reason
            start
        }
    }
}</code></pre>
</div>
</div>

The client needs to make sure to always add code that handles unknown <code>__typename</code> values, so new errors can be added to the Union without breaking anything. When we add an <code>OrderCancelled</code> response, the client can see that there's a new value for the <code>__typename</code>, which they probably have to add special handling for.

(Side note, just for the sake of completeness: a very similar solution can be achieved with GraphQL Interfaces; but I won't go into details, as it doesn't add a lot to the argument.)

For nested errors, the result Types can be nested, too:
<div class="listingblock">
<div class="title">Listing 6. Nested CustomerResult Union</div>
<div class="content">
<pre class="highlight"><code class="language-graphql" data-lang="graphql">type Order {
    id: ID
    orderDate: Date
    customer: CustomerResult
}
union CustomerResult = Customer | CustomerBlocked
type CustomerBlocked {
    positive: Boolean
}</code></pre>
</div>
</div>

<h3 id="_one_way_street">One Way Street</h3>

The Union based solution is great: the Schema clearly documents what possibly can go wrong, business-wise, and what alternative data is available. But it doesn't mitigate one major problem: communication is one-way, server to client. The backend still doesn't see which clients handle which errors, as clients can react to any of the possible <code>__typename</code> values without selecting even a single detail field, e.g. the <code>OrderNotFound</code>. One could say that the <code>__typename</code> is just the new <code>code</code> extension made part of the Schema, which is a major step, but the backend can't see that some error is not needed any more and can be removed. Or that a specific client has started to handle a specific business error. The concept, as clean as it is, is somewhat impure around the edges. It regulates the communication from the server to the client, but doesn't improve the communication of the requirements from the client to the server, which is a central benefit of GraphQL.

<h2 id="_result_fields">Result Fields</h2>

In order to make the clients' requirements visible, one option is to declare the <code>OrderResult</code> from above to be a regular wrapper Type instead of a Union, i.e. allow clients to select what business errors they want to handle:
<div class="listingblock">
<div class="title">Listing 7. OrderResult Wrapper</div>
<div class="content">
<pre class="highlight"><code class="language-graphql" data-lang="graphql">type OrderResult {
    order: Order
    error_orderLocked: OrderLocked
    error_orderNotFound: OrderNotFound
}</code></pre>
</div>
</div>

Normally, exactly one of these fields is set, which is, sadly, not visible in the Schema; it must be documented as a convention.

If an error situation arises that a client didn't expect, e.g. they don't know that orders can be locked, they don't select the error field. In order to prevent the situation that the <code>order</code> field is <code>null</code> and the client has no idea why, we fall back to reporting the error via the classic, technical <code>errors</code> field. This allows us to add new business errors at any time, e.g. when we add the feature of cancelling orders, we add an <code>error_orderCancelled</code>; and existing clients get it as technical <code>errors</code>.

If it's still possible to retrieve the <code>order</code> data and the new error is not critical, i.e. existing clients can safely ignore the fact that an order is, e.g., locked while reading, we can simply <em>not report</em> the error.

In both cases, if clients want to react to a new error, they can simply select the corresponding new error field. And on the other hand, if we see that no client selects an error field anymore, we can safely remove it.

Nesting works exactly like with the Union responses described above.

<h3 id="_breaking_changes">Breaking Changes</h3>

But introducing such a result wrapper Type (also if it's a Union) is a breaking change. We can't do that as long as clients still expect us to return the data directly. We could use the wrappers right from the beginning, every time, even before we know if we might need to add business errors later. But even a scalar field could need to be in a business error state: e.g. just the customer name could be blocked, i.e. we'd have to wrap it in a <code>CustomerNameResult</code>. Doing so for every field doesn't make sense.

So we need a strategy to <em>migrate</em> existing queries, define a new field with a new name returning the new wrapper type, while the old field still returns the flat data. If the old field is not selected anymore, we can remove it. To have a standard naming convention for the wrapped fields, we could suffix the name with a word, e.g. append <code>Result</code> to the old <code>order</code> field to create <code>orderResult</code>. But this can easily be confusing, e.g. a mutation <code>createGame</code> would become <code>createGameResult</code>, which sounds like something very different. So I'd opt for a single underscore, e.g. <code>order_</code>; but that's not explaining itself. And would we do that also for fields where we do know from the beginning that they have some error states? It's not easy to make such an API consistent.

<h2 id="_flat">Flat</h2>

To prevent breaking changes by design, we could add the error fields directly to the return Type:
<div class="listingblock">
<div class="title">Listing 8. Order With Error Fields</div>
<div class="content">
<pre class="highlight"><code class="language-graphql" data-lang="graphql">type Query {
    order(id: String): Order
}
type Order {
    id: ID
    orderDate: Date
    # ...
    error_orderLocked: OrderLocked
    error_orderNotFound: OrderNotFound
}</code></pre>
</div>
</div>

Again, if some error occurs that the client didn't select, we fall back to reporting it like a technical error.

Nesting works naturally, i.e. the <code>Customer</code> type could have an <code>error_customerBlocked</code> field.

At first glance, this solution looks perfect, but it has a major drawback: all data fields must be nullable.

<h2 id="_extra_queries">Extra Queries</h2>

GraphQL allows us to issue several queries at once; so we can add separate queries for the business errors. In order to link them to the original query, we use a naming convention <code>&lt;query-path&gt;_error_&lt;code&gt;</code>:

<div class="listingblock">
<div class="title">Listing 9. Order and extra error fields</div>
<div class="content">
<pre class="highlight"><code class="language-graphql" data-lang="graphql">type Query {
    order(id: String): Order
    order_error_orderLocked: OrderLocked
    order_error_orderNotFound: OrderNotFound
    order_customer_error_customerBlocked: CustomerBlocked
}</code></pre>
</div>
</div>

An actual query could be:
<div class="listingblock">
<div class="title">Listing 10. Order and extra error field query</div>
<div class="content">
<pre class="highlight"><code class="language-graphql" data-lang="graphql">query order($id: String) {
    order(id: $id) {
        id
        orderDate
        customer {
            name
        }
    }
    order_error_locked {
        reason
        start
    }
    order_customer_error_customerBlocked {
        reason
    }
}</code></pre>
</div>
</div>

Mangling the query path into a field name may feel messy, but it works.

Also here, if some error occurs that the client didn't select, we fall back to reporting it like a technical error.

Note that the return type of <code>order</code> must be nullable now.

<h2 id="_conclusion">Conclusion</h2>

So these are the five options we have, and all have their merits:
<table><colgroup> <col style="width: 20%;" /> <col style="width: 20%;" /> <col style="width: 20%;" /> <col style="width: 20%;" /> <col style="width: 20%;" /> </colgroup>
<thead>
<tr>
<th>Option</th>
<th>Visible in Schema (server→client)</th>
<th>Visible in Query (client→server)</th>
<th>Extensible
(no breaking changes)</th>
<th>Side effects</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="#_technical_errors">Technical Errors</a></td>
<td>𐄂</td>
<td>𐄂</td>
<td>✓</td>
<td></td>
</tr>
<tr>
<td><a href="#_result_union">Result Union</a></td>
<td>✓</td>
<td>𐄂</td>
<td>𐄂</td>
<td></td>
</tr>
<tr>
<td><a href="#_result_fields">Result Fields</a></td>
<td>(✓)</td>
<td>✓</td>
<td>𐄂</td>
<td></td>
</tr>
<tr>
<td><a href="#_flat">Flat</a></td>
<td>(✓)</td>
<td>✓</td>
<td>✓</td>
<td>All data fields must be nullable</td>
</tr>
<tr>
<td><a href="#_extra_queries">Extra Queries</a></td>
<td>(✓)</td>
<td>✓</td>
<td>✓</td>
<td>The data field must be nullable</td>
</tr>
</tbody>
</table>
<code>(✓)</code> means that, while the business errors are visible in the Schema, there is an extra convention on top, that you'll need to know in order to truly understand the Schema.

No option is perfect, so the question remains: which one should I choose? Is it just a matter of taste? The elegance in the Schema when using Unions is compelling, but the drawbacks are severe. I'm especially hesitant to prematurely introduce 'best practices' like result wrapper Types; I've already had too many best practices proven to not actually be best in all cases. My personal conclusion is that using Flat error fields provides the best balance between extensibility and visibility.

What do you think? Let's discuss!

GraphQL business error responses revisited

RFC-7807 problem details with Spring Boot and JAX-RS

During a discussion about how to best implement our domain objects when applying <a href="https://en.wikipedia.org/wiki/Domain-driven_design" target="_blank" rel="noopener noreferrer">DDD</a>, which has recently become more and more popular, a colleague pointed me to Martin Fowler's article about <a href="https://www.martinfowler.com/bliki/AnemicDomainModel.html" target="_blank" rel="noopener noreferrer">Anemic Domain Models</a> (2003). Martin is one of my heroes; I try to read as many of his writings and watch as many of his talks as I can, because they give me deep insight and a balanced understanding of all options available, which I consider essential to making smart software design decisions <sup>[<a id="_footnoteref_1" title="View footnote." href="#_footnotedef_1">1</a>]</sup>. In the aforementioned article, Martin uses rather opinionated language to say that the Anemic Domain Model is an anti-pattern 'robbing yourself blind'. So he must have (or at least must have had) good reasons to be so strongly opposed to it; just as Eric Evans, whom Martin cites several times.

Now, in 2019, I have difficulties seeing it as universally problematic, even after rereading the article several times. So I think I should maybe add my two cents, even if it may upset the DDD community.

<h2 id="user-content-what-is-an-anemic-domain-model">What are Anemic Domain Models?</h2>

In Martin's seminal <a href="https://martinfowler.com/books/eaa.html" target="_blank" rel="noopener noreferrer">P of EAA</a> book (2002), a Domain Model is defined as 'an object model of the domain that incorporates both behavior and data'. This clearly sets it apart from Entity Objects, which are object representations of only the <em>data</em> stored in a database (relational or not), while the <em>behavior</em> is located in separate classes instead. Note that this divide is not really a layering, it’s just procedures working with pure data structures. This is the only way to work in procedural languages like C.

So putting the different terminology aside and simplifying things, an Anemic Domain Model is basically just another word for using pure-data Entity Classes, with the behavior extracted to what we might call Operation Classes <sup>[<a id="_footnoteref_2" title="View footnote." href="#_footnotedef_2">2</a>]</sup>.

<h2 id="user-content-whats-wrong-with-pure-entities-operations">What’s wrong with pure Entities + Operations?</h2>

In the article, Martin acknowledges that, while true, it’s not sufficient to just declare that the Entity/Operation dichotomy is not OO, while Domain Models are. The main pain he sees is the 'awkwardness of mapping to a database'; he views this as only worthwhile if the objects being mapped to actually utilize their potential.

Two factors may have invalidated his premise, though: (a) ORM tools like JPA have improved considerably; you can also directly store objects and its dependencies as a document into e.g. MongoDB. It’s actually much <em>easier</em> as well as cleaner to simply map an Entity to/from a database than to work directly with e.g. JDBC result sets, or manually map the data. (b) Much more importantly, nowadays the DB model is generally controlled by a single application only; luckily teams mostly stopped integrating via databases by sharing their tables with other teams. So if the domain model needs to evolve, the DB schema can be migrated without any coordination ceremony; and we can work with a unified, or even ubiquitous, model <sup>[<a id="_footnoteref_3" title="View footnote." href="#_footnotedef_3">3</a>]</sup>.

Nonetheless: having the business methods directly on the business objects has an indisputable elegance; you should not push that aside too easily.

<h2 id="user-content-can-entities-be-real-domain-objects">Can Entities <em>be</em> real Domain Objects?</h2>

Bean validation is a natural fit to add constraints to your Entities: you can simply add, for instance, a <code>@Past</code> annotation to the <code>LocalDate born</code> field of a <code>Person</code> entity and the validation is automatically done whenever you want to store it. This can be seen as an extension to the type system you use for e.g. <code>LocalDate</code>. No extra code, pure domain logic: nice.

You can also create custom constraints, e.g. <code>@Name</code> to prevent smileys and other inappropriate characters in people's names; the validator you’ll need to write is also pure domain logic code. And bean validators can deal with substantial complexity: you can (a) validate multiple fields in combination; (b) use validation groups, e.g. for different states of the entity; and (c) inject helper classes for repositories and others. This allows you to check, for example, that the real estate agent has the legally required certifications to sell a specific building, depending on its location and other factors. Pure domain logic: nice.

JPA Entity classes can also have additional domain methods, e.g. our <code>Person</code> class can have a method <code>getAge</code> that calculates the current age of the person from the <code>born</code> field; or creating a <code>Customer</code> object may automatically set the first registration date. Pure domain logic: nice.

Some domain methods require repositories or other helpers; e.g. when (re)calculating the sum of an order, we may have to fetch the article prices. We could inject all the helpers we needed into our entities <sup>[<a id="_footnoteref_4" title="View footnote." href="#_footnotedef_4">4</a>]</sup>; but it’s a good Clean Code practice to move complex interactions out of an Entity into an Operation class; esp. if multiple entities are involved; testing gets much easier in this way, too.

Note that these Operation Classes are still part of the Domain Model: they should be pure domain logic and probably reside in the same package as the entities. Only the invocation is different: instead of calling <code>order.calculateTotalPrice()</code>, you have an <code>OrderOperations</code> class where you call <code>calculateTotalPrice(order)</code> <sup>[<a id="_footnoteref_5" title="View footnote." href="#_footnotedef_5">5</a>]</sup>. I think that’s not too bad.

<h2 id="user-content-application-logic">Application Logic</h2>

Fully blown business transactions sometimes get seriously complex. For example, closing a sale of an insurance may have to trigger many other processes; we may even have to span multiple Bounded Contexts. This is often rightfully extracted into a separate application layer (some call it Service Layer) that 'only coordinates tasks and delegates work'; you may call it a <a href="https://www.martinfowler.com/eaaCatalog/transactionScript.html" target="_blank" rel="noopener noreferrer">Transaction Script</a>. But the line between core domain and application logic is often blurry and different people draw it at different locations.

When building a <a href="https://samnewman.io/patterns/architectural/bff/" target="_blank" rel="noopener noreferrer">Backend For Frontend</a> where the frontend has to display additional data, e.g. the ratings of an article when adding it to an order, that’s clearly application logic; or a multi-step creation operation, where every step should be persisted, is also clearly application logic: the decisions are about <a href="https://en.wikipedia.org/wiki/User_experience" target="_blank" rel="noopener noreferrer">UX</a>, which is completed by the visual decisions that I’d define to be a presentation layer: a web application often differs from a desktop application in how it looks, but it probably has the same application logic <sup>[<a id="_footnoteref_6" title="View footnote." href="#_footnotedef_6">6</a>]</sup>; the application logic can easily be unit-tested with <a href="https://www.martinfowler.com/bliki/SubcutaneousTest.html" target="_blank" rel="noopener noreferrer">subcutaneous tests</a>, while testing the presentation requires serious integration setup.

Martin cites Eric Evans' statement that the application layer 'does not contain business rules or knowledge'. But isn’t the coordination of the processes involved in a business transaction also business domain knowledge? Doesn’t it implement business rules? I find the distinction between application and domain logic difficult to draw out of thin air. I prefer to postpone the creation of design artifacts like layers until a specific need arises.

Separating data from behavior triggers memories of the old-school thinking of databases defining the constraints, and fat frontend applications doing the actual work. I can understand why OO purists like Martin disregard this approach. But it also has its strengths: when we separate only complex behavior, it may not be clear where we should look for a specific operation; we’d need additional conventions to define when we do one thing and when we do the other. When we separate <em>all</em> the behavior — validation logic as well as calculations and complex operations — from the data, things are more uniform and may be easier to grasp …​ and to test!

And it’s not at all old-school: when brought to an internet scale, it may result in things like the <a href="https://solid.inrupt.com" target="_blank" rel="noopener noreferrer">SOLID</a> project, where your personal data (pictures, blogs, etc.) is strictly separated from the applications you use to work with it (Facebook, Twitter, etc.), so you can easily switch to other applications without losing your data. This is not only about re-implementation of the same type of application; it’s about different types of applications working on the same data. This would not be possible without an Anemic Domain Model.

<h2 id="user-content-conclusion">Conclusion</h2>

I agree that it’s a <a href="https://martinfowler.com/bliki/CodeSmell.html" target="_blank" rel="noopener noreferrer">smell</a> to prematurely start with an Anemic Domain Model and to declare this as a best practice; but I disagree that OO purity is a hard requirement for qualifying as a Domain Model. It’s a valid solution to separate Operation Classes from Entity Classes; just keep them in the same Domain Model layer! Sometimes there are compelling reasons to separate logic from data; and sometimes it’s better not to. Software developers must be (en)able(d) to see for themselves when it’s better to do one or the other. And they should know how to use bean validation, for example.

Naming things is one of the <a href="https://www.martinfowler.com/bliki/TwoHardThings.html" target="_blank" rel="noopener noreferrer">hard things in Computer Science</a>, and it’s truly important; one could argue that we should not call something a Domain Model when the objects are anemic. But saying that DDD only works with a pure OO paradigm excludes and denies not only the pragmatic approaches described above, but also, e.g., functional programming; and that would be a big loss. I think that an Anemic Domain Model still qualifies to be called a proper Domain Model — and it’s good to have it as a valid tool in your belt.

What do <em>you</em> think? Am I completely wrong? Please comment!<br>

<hr />

<div id="_footnotedef_1"><a href="#_footnoteref_1">1</a>. I don’t like the connotations of the term 'architecture'.</div>
<div id="_footnotedef_2"><a href="#_footnoteref_2">2</a>. Different schools of software design overloaded a lot of terms with quite different semantics. So picking one without causing too much confusion is really difficult.</div>
<div id="_footnotedef_3"><a href="#_footnoteref_3">3</a>. While annotations like JPA <code>@Entity</code> are not really part of a pure Domain Model, they are not hard runtime dependencies either: you can simply ignore them in tests; and what most developers don’t know, even when the annotations are not on the classpath, classes using them load just fine! So it’s basically a matter of taste: do you dislike such a 'pollution'? Or do you shun the extra mapping code required to make it 'pure'? Purity doesn't directly add value; the mapping code qualifies to be called anemic itself. Some NoSQL databases can even store Entity classes that have no annotations at all, but real DDD purists may still not like the fact that annotations <em>might</em> become necessary one day.</div>
<div>If DDD was taken utterly strict, then Domain Models could only be defined in a DSL or a modeling language. Using a general programming language like Java and the core JDK libraries <em>does</em> establish a dependency on a technical infrastructure. Just take the JAXB annotations, that recently where removed from the JDK. I’m in the more pragmatic camp and would not only allow the language and the core libraries. I accept also frameworks; at least if they are as stable as open standards like JPA or even more bean validation. Martin also states that putting 'data source logic …​ in a domain object …​ is orthogonal to [his] view of anemia,' so the discussion may be important, but not about this article.</div>
<div id="_footnotedef_4"><a href="#_footnoteref_4">4</a>. Depending on your tech stack, this may not work out of the box, so you may have to add, e.g., a persistence listener and inject the helpers manually; also constructor injection is getting cumbersome, if you want all the data fields in the constructor, too.</div>
<div id="_footnotedef_5"><a href="#_footnoteref_5">5</a>. You could add a method to the Entity that only delegates to a hidden helper. But then you’d still have to test this via the entity method. Testing would not be any easier and the other problems would remain. Testing the helper separately would violate the principle to test only the public behavior in exactly the same way as a client would use it.</div>
<div id="_footnotedef_6"><a href="#_footnoteref_6">6</a>. As Sam Newman points out in his BFF article: you should design mobile small-screen applications as a distinct application, when they require different patterns of interaction... which they often do.</div>

DDD vs. Anemic Domain Models (Martin Fowler)

Structured JUnit 5 testing

IT Consulting Expert

„Kommunikation“ ist das Passwort für gute Projekte

Kick-start your microservice project with JHipster

CloudWatch on AWS: How to tackle high-security requirements

Senior IT Consultant

When it comes to building applications in WebAssembly, Rust and C++ are the two most <a href="https://blog.scottlogic.com/2022/06/20/state-of-wasm-2022.html" target="_blank">frequently used</a> languages at the moment. However, both languages have relatively steep learning curves, so it makes sense to also consider other options. Go is known for its simplicity and easy-to-use concurrency model. This makes it a great choice for a wide range of web oriented applications. In this article, we will explore how Go applications can target WebAssembly and what the interaction between JavaScript and Go looks like.

## The example application
While <a href="https://webassembly.org/" target="_blank">WebAssembly</a> has seen quite some hype in recent years, it seems that most blogs only focus on very simple applications, such as "hello world" or summing two integers. In my experience with WebAssembly in C++, the real fun only begins when working with dynamic memory, such as arrays or objects. However, we will see that Go handles even these cases very well.

![sobelexample.png](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/vtzqOPAT66Rt8zVRoKiC)
<p style="text-align: center;">The example application. Source image on the left by <a href="https://commons.wikimedia.org/wiki/File:Shanghai_skyline_waterfront_pudong_5166168_69_70.jpg">Ermell</a>.</p>

We will study a small example application for edge detection in images. The application makes use of the JavaScript File API to load an image file as a byte array. The byte array is sent to a WebAssembly module written in Go. The module applies a convolution filter and returns another image, which the JavaScript code will display with an `<image>` element.

## Basic application structure
Let us start with the basic scaffolding for a Go WebAssembly application:
```go
package main

import "syscall/js"

func applySobelOperator(this js.Value, args []js.Value) any {
	// TBD
}

func main() {
	js.Global().Set("applySobel", js.FuncOf(applySobelOperator))
	<-make(chan bool)
}
```
We aim to expose the function `applySobelOperator` to JavaScript. The Go `syscall/js` package provides support for this interaction. The type of the function must be:
```go
func(this js.Value, args []js.Value) any
```
This function type represents a JavaScript function invocation with the static type system of Go. The first parameter is `this`, which will hold whatever the function was bound to in JavaScript. The second parameter, `args`, is an array of the arguments, because in JavaScript, an arbitrary number and types of arguments can be passed to a function. We will see below how to work with those values. The function may return any Go type. 

When the main method is executed, the WebAssembly instance will register the function in the global object of JavaScript, i.e. `window` when running in the browser. The function can be invoked from JavaScript as long as the instance is still running. The easiest way to let the main method run forever is by waiting on an anonymous channel.

Go can be compiled to WebAssembly with the standard Go compiler, using
```bash
$ GOOS=js GOARCH=wasm go build -o sobel.wasm .
```
Additionally, we have to serve the `wasm_exec.js` bundled with the Go release. The file is the JS counterpart to the `syscall/js` package. It provides a few JS implementations of this package and handles transferring data from/to the WebAssembly instance's memory. This script has to be executed before we can run the WebAssembly module.
```bash
$ cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" wasm_exec.js
```

We run the `.wasm` file from JavaScript like so:
```html
 
<body>
	<div class="content">
		 
	</div>
	<script src="wasm_exec.js"></script>
	<script>
		const go = new window.Go();
		
		WebAssembly.instantiateStreaming(
			fetch("sobel.wasm"),
			go.importObject
		).then(
			(obj) => {
				go.run(obj.instance);
			}
		);
	</script>
</body>
 
```

Already with those small snippets of Go and HTML/JS, quite a few things are happening at runtime:
1. Browser loads and renders the `.html`.
2. The `wasm_exec.js` is loaded and executed. It adds its functionality in a global class `Go`. 
3. The `<script>`-tag is executed, this will
4. fetch the `.wasm` file containing the WebAssembly byte code.
5. `WebAssembly.instantiateStreaming` will compile and instantiate the byte code into a `WebAssembly.Instance`.
6. The instance is run, where
7. our Go main function registers `applySobel` in `window`, ready to be called.

## Interaction between Go and JavaScript
For loading an image from disk and handing it over to the WebAssembly instance, we implement the `change` event on an `<input type="file">` element:
```javascript
fileInput.addEventListener("change", handleFiles);

async function handleFiles() {
	if (this.files.length) {
		const file = this.files[0];
		const arrayBuffer = await file.arrayBuffer();
		const input = new Uint8Array(arrayBuffer);
		const result = window.applySobel(input);
		targetImage.src = URL.createObjectURL(new Blob([result]));
	}
}
```
The `arrayBuffer()` method on the selected file already returns the raw binary data. We wrap it in a typed array view, to make it compatible with our Go code. 

A central type defined in the `syscall/js` package is `js.Value`. This type can represent any JavaScript value. It provides methods to retrieve typed values, such as `Int()`. In case our compile-time assumption on the type of the value is wrong, e.g. we call `Int()` on a floating point value, the program crashes at runtime. 
If the value is a JS object, it is also possible to call methods with `Call(m string, args ...any)`, or retrieve properties with `Get(p string)`. 
Putting those together, we retrieve the length of the parameter, the byte array, with:
```go
args[0].Get("length").Int()
```
Go also has support for JS <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays" target="_blank">typed arrays</a>. This is quite handy, because it allows us to transfer raw data from and to JavaScript.
Using `CopyBytesToGo(dst []byte, src js.Value)`, we can copy the content of an `Uint8Array` into a slice of type `byte`.
It is also possible to create JavaScript objects, e.g. a new `Uint8Array` of a certain size with:
```go
js.Global().Get("Uint8Array").New(size)
```
Creating a JavaScript object is especially useful when we want to return it from a function.

Here is the implementation of the Go function called in the above JavaScript extract:
```go
func applySobelOperator(this js.Value, args []js.Value) any {
	inputBuffer := make([]byte, args[0].Get("byteLength").Int())
	js.CopyBytesToGo(inputBuffer, args[0])
	img, _, _ := image.Decode(bytes.NewReader(inputBuffer))
	
	resultImage := sobelConvolution(img)
	
	var outputBuffer bytes.Buffer
	png.Encode(&outputBuffer, resultImage)
	outputBytes := outputBuffer.Bytes()
	
	size := len(outputBytes)
	result := js.Global().Get("Uint8Array").New(size)
	js.CopyBytesToJS(result, outputBytes)
	
	return result
}
```

The actual implementation of the sobel image convolution is omitted for brevity. The full implementation is available <a href="https://github.com/j-blue-arz/sobel-wasm" target="_blank">here</a>.

## Conclusion
Using Go to build a WebAssembly module is straight-forward. No separate toolchain is required. The standard library brings a package to handle interop between JavaScript and Go.

This power allows some flexibility on how to structure the application. For example, it would have been also possible to write the entire application in Go and perform DOM manipulation by means of something like 
```go
js.Global()
	.Get("document")
	.Call("getElementById", "target")
	.Set("property", value)
```
Another option would be to use the Canvas API to extract the raw RGBA channel of the image and only transfer that to WebAssembly. 

However, the great interoperability between Go and JavaScript comes at a cost: Our Go module is dependent on JavaScript, which means it will not run in a non-JavaScript environment. That is what <a href="https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/" target="_blank">WASI</a>, the WebAssembly System Interface, is about. With WASI, WebAssembly has the potential to alter the way we write and deploy applications. Go not being compliant with WASI is a big drawback. In an upcoming article, we will see how to respond to this and other challenges when using Go for WebAssembly.

Targeting WebAssembly with Go

WebAssembly was initially created to bring languages other than JavaScript to the browser. Its design goals include portability, safety and performance. WASI (**W**eb**A**ssembly **S**ystem **I**nterface) lifts those capabilities to the world outside the browser. This article explores some possible use cases and shows a simple example in which WASI enables running the same binary in three different environments.

## Wait, what? WASI? What is that?

WASI, the WebAssembly System Interface, aims to standardize how WebAssembly modules interact with the outside world. 

WebAssembly, abbreviated *wasm*, is a compilation target for programming languages. It was designed to bring languages like C++ or Rust to the browser. Today it supports a multitude of other languages, and it is not limited to the browser anymore.

Each WebAssembly module runs in a sandboxed environment. It exposes functions to be called from its host, but it cannot access the host by itself. WebAssembly does not have any built-in functionality. All system calls which are normally provided by a standard library, such as reading from or writing to a file, retrieving the system clock, or generating random numbers, have to be injected by the host.

WASI provides an interface specification for these common functions. WASI-compliant runtimes like <a href="https://wasmtime.dev/" target="_blank">Wasmtime</a>, <a href="https://wasmer.io/" target="_blank">Wasmer</a>, or <a href="https://github.com/WasmEdge/WasmEdge" target="_blank">WasmEdge</a> make it possible to run WebAssembly embedded in other languages, across a variety of platforms such as desktop, browser, mobile, or IoT. Furthermore, WASI already enables WebAssembly support to be built-in into <a href="https://www.singlestore.com/blog/revolution-2022-code-engine-for-wasm/" target="_blank">databases</a>, <a href="https://wasmlabs.dev/articles/apache-mod-wasm/" target="_blank">web servers</a>, and <a href="https://www.docker.com/blog/docker-wasm-technical-preview/" target="_blank">container runtimes</a>.

WASI allows building applications in any language that compiles to WebAssembly and run them in any place where a runtime is available.

## Use Cases

WASI allows for a lot of interesting use cases:

**Portable business logic**. When developing applications for a large and diverse user base with multiple supported devices, we are faced with the challenge of code duplication across the different clients. With WASI, we can write business logic or a complicated subsystem only once, compile to WebAssembly and embed it in all client implementations. WASI runtimes such as Wasmtime provide support for many kinds of clients, from browsers over mobile phones to IoT devices. Compared to similar solutions, like the JVM or JavaScript, WebAssembly offers predictable performance, which makes it a better target for applications in domains like Multimedia, AR/VR, or CAD. The BBC <a href="https://medium.com/bbc-product-technology/building-a-webassembly-runtime-for-bbc-iplayer-and-enhanced-audience-experiences-7087455808ef" target="_blank">for example</a> has used WebAssembly for their multimedia offerings.

**Cross-language platform**. Assume you are writing an application in Go. You want to integrate a specific ML/AI library from the Python ecosystem. But also a Rust-based high performance image manipulation library. And maybe a certified implementation for a security-relevant algorithm written in C. You are faced with the task to transpile or port the existing 3rd party codebases. Or you call them via a language binding, with the challenge that it has to work for all operating systems and environments. WASI provides a convenient solution to this challenge. All those libraries could be (pre-)compiled to WebAssembly, and integrated with a WASI runtime. From that perspective, WASI might transform WebAssembly into a universal cross-language application platform. The WebAssembly package manager, <a href="https://wapm.io" target="_blank">wapm</a>, could facilitate this vision.

**Lightweight containers**. Container runtimes such as containerd (Docker) let us run isolated container workloads by providing OS-level virtualization. WebAssembly allows sandboxed, isolated execution of applications. So an alternative to packaging an application in a container image could be to instead compile it to WebAssembly. Compared to running OCI containers, WebAssembly offers faster startup and lower disk footprint. This can be a great advantage for highly scalable microservice architectures, but also for serverless offerings of cloud providers. <a href="https://github.com/containers/crun" target="_blank">Crun</a> and <a href="https://www.docker.com/blog/build-share-run-webassembly-apps-docker/" target="_blank">containerd</a> already support WebAssembly, and Azure Kubernetes has <a href="https://aka.ms/aks/wasi-nodepool" target="_blank">preview support</a> for WASI workloads.

## An example with Go

We build a WebAssembly application and run it in three different environments. Let's say we are interested in prime number generation. The Go standard library includes a probabilistic primality check for arbitrary big integers. So checking if a number is prime is just a few lines in Go. We build a WebAssembly binary and run the same application from the command line, in a web browser, and from GraalVM.

In a <a href="https://blog.codecentric.de/webassembly-with-go" target="_blank">previous blog article</a>, we already saw how to compile a Go application to WebAssembly. The article highlighted the great interoperability between Go and JavaScript, but also mentioned that Go does not support WASI. However, there is an alternative Go implementation, <a href="https://tinygo.org/" target="_blank">TinyGo</a>, which does comply with WASI.

### TinyGo to the rescue

TinyGo was initially created for embedded environments. In recent times, it has also become the go-to compiler for Go WebAssembly projects. Not only does it support WASI, it also solves another well-known problem of Go: its big binary size. Even for small programs, Go application binaries total in about 3 MB, due to the embedded Go runtime. This is not so much of a problem for CLI applications or microservices running on the server, but can be a challenge for application files distributed over the web. TinyGo produces WebAssembly binaries by a factor of 3-5 smaller than the standard Go compiler. The example application below can be brought down to about 350 kB with a few reasonable compiler flags.

TinyGo should be the first consideration for Go applications targeting WebAssembly. However, it has some drawbacks. Besides the already mentioned lack of implemented WebAssembly features, it also lags behind a few months in new Go language features. Also some parts of the standard library, e.g. reflection, are not supported. This can be a problem when we have no or little control over the code we want to compile, e.g. when compiling 3rd party libraries.

Below is the Go application source code for our primality check. It is written like a normal, runnable CLI-application. We do not import the `syscall/js` package, in contrast to a Go Webassembly application targeting the browser. The input is a command line argument and the program outputs directly to the standard output (stdout). 

```go
package main

import (
    "fmt"
    "math/big"
    "os"
)

func main() {
    number := new(big.Int)
    fmt.Sscan(os.Args[1], number)
    if number.ProbablyPrime(10) {
        fmt.Println(os.Args[1] + " is probably prime\n")
    } else {
        fmt.Println(os.Args[1] + " is not prime\n")
    }
}
```

This method, writing a runnable CLI application, is typical for WASI applications. The other option of targeting WebAssembly and WASI would be to create a library exporting multiple functions. However, as of today, these library functions can only accept WebAssembly native types, that is, integers or floats. The CLI style greatly simplifies handling strings, which is the reason we opt for it here.

With TinyGo installed on our system, we can compile the above code to WebAssembly with

```shell
$ tinygo build -o prime.wasm -target wasi main.go
```


### From the command line

After installing Wasmtime, a popular WebAssembly runtime, we can run our prime.wasm binary like this:

```shell
$ wasmtime prime.wasm 359334085968622831041960188598043661065388726959079837
359334085968622831041960188598043661065388726959079837 is probably prime
```
This works both in Windows and in Linux.


### Running in browser

All browsers have built-in support for loading, compiling and running WebAssembly binaries. However, when targeting WASI, the WebAssembly module requires a few imports which have to be satisfied. For example, the module will need functions called `args_sizes_get` and `args_get` to retrieve the command line arguments. It is possible to <a href="https://www.wasmtutor.com/webassembly-barebones-wasi" target="_blank">provide these imports manually</a>, but it is far easier to use a precompiled runtime, such as wasmer-js. Wasmer-js is a thin JavaScript wrapper over the wasmer runtime, which is implemented in Rust. 

```javascript
import { init, WASI } from '@wasmer/wasi'

let module = undefined;

export async function computePrime(number) {
    if (module === undefined) {
        await init();
        module = await WebAssembly.compileStreaming(fetch('prime.wasm'));
    }

	let wasi = new WASI({
        env: {},
        args: [
            'prime.wasm', number
        ,
    });
    await wasi.instantiate(module, {});
    wasi.start();
    return wasi.getStdoutString();
}
```

In line 8 of the above JavaScript source code, we use the browser API to download and compile the `prime.wasm` binary to a WebAssembly module. The rest of the code instantiates and runs this module, providing the input for the primality check. 

### Embedded in Java

In general, the integration between Java and WebAssembly had a rough start so far. For example, projects aiming to compile Java to WebAssembly face a few difficult challenges. One is that WebAssembly does not have garbage collection built-in. The other is that Java supports reflection, i.e. it allows to evaluate and manipulate properties of a program at runtime. This is close to impossible for compiled languages such as WebAssembly. 

Calling WebAssembly from Java is also not well supported. Besides a few java libraries (e.g. <a href="https://extism.org/docs/integrate-into-your-codebase/java-host-sdk/" target="_blank">Extism</a>), there is one notable exception, which we will explore here: the Java runtime GraalVM. GraalVM is among other things known for its Polyglot API, allowing it to run other languages like Python, R, and Ruby. And also WebAssembly.
Here is how we run our `prime.wasm` binary from Java in GraalVM:

```java
public static void main(String[] args) throws Exception {
    File file = new File(Thread
            .currentThread()
            .getContextClassLoader()
            .getResource("prime.wasm")
            .getFile());
    Source.Builder sourceBuilder = Source.newBuilder("wasm", file);
    Source source = sourceBuilder.build();

    Context.Builder contextBuilder = Context.newBuilder("wasm")//
            .option("wasm.Builtins", "wasi_snapshot_preview1").//
            arguments("wasm", new String[]{"", args[0]});

    try (Context context = contextBuilder.build()) {
        context.eval(source);

        Value mainFunction = context
                .getBindings("wasm")
                .getMember("main")
                .getMember("_start");
        mainFunction.execute();
    } 
}
```

Embedding WebAssembly in GraalVM requires a bit more effort than in JavaScript. This is mainly due to the Polyglot API, which is not WebAssembly specific, but aims to represent all embedded languages with the same programming interface. We first have to represent the `prime.wasm` file as a `Source` object (line 8). Here, it is loaded as a file, but it is also possible to work with byte streams, if we were to e.g. load the binary from the network.

Then we create a `Context` object for the language `"wasm"`. We activate WASI for this context in line 11. Among lots of other things, this will inject `System.out` as the standard output for the WebAssembly application.

Our Go application accesses the second command line argument as the input number, because the first argument of CLI application is always reserved for the program name. Java does not have this convention, so we shift the argument array by one entry in line 12.

Line 15 will parse the code and create a runnable application, for which we then execute the main method. Given that GraalVM is installed, we can run the above with

```shell
$ javac Prime.java
$ java Prime 359334085968622831041960188598043661065388726959079837
359334085968622831041960188598043661065388726959079837 is probably prime
```

The full Go, JavaScript, and Java source code of these examples is available [here](https://github.com/j-blue-arz/go-prime-wasi/).

## Conclusion

WebAssembly and WASI have gained a lot of attention in the last few years. In my opinion, together they have the potential to alter the way we write and deploy applications: as an example, they allow us to build applications once and run them in a wide variety of different environments, as we have seen in this blogpost.

A number of big players, for example Mozilla, Docker, <a href="https://www.fastly.com/blog/how-fastly-and-developer-community-invest-in-webassembly-ecosystem" target="_blank">Fastly</a>, and <a href="https://www.redhat.com/en/blog/red-hat-and-webassembly" target="_blank">Red Hat</a>, have understood the potential and are investing in the WebAssembly community.

Also, an increasing number of developers are using WebAssembly and WASI for a growing set of use cases: <a href="https://blog.scottlogic.com/2022/06/20/state-of-wasm-2022.html" target="_blank">The State of WebAssembly 2022 survey</a> shows considerable increase in usage in serverless environments, containerization, and also plugin-development. This last topic, plugin-architectures, makes use of the security guarantees of WebAssembly to integrate untrusted third-party plugins in a secure environment. <a href="https://www.infoq.com/presentations/webassembly-server-side/" target="_blank">Shopify</a> is successfully using such an architecture to allow customer adaptions to their checkout process. In an upcoming article, I will have a deeper look into this use case.

What are you doing with WebAssembly? I'd love to hear about your use cases, challenges, and ideas. Feel free to contact me!

Compile once, run anywhere with WebAssembly and WASI

Plug-in architectures are an essential concept for developing customizable software. In a plug-in architecture, the application logic is split into a host (or core) system and a number of plug-in components. These plug-ins enable customers to tailor the business logic of the application to their specific use case. Plug-in architectures are widely used in end-user applications or self-hosted software products. Examples of these include integrated development environments (IDEs), web browsers, content management systems (CMS), graphic design software, and many others.

However, the use of this architecture can pose a challenge when the software system to be customized is deployed on a central server. This is typically the case in Software as a Service (SaaS) applications. In this case, implementing a plug-in architecture requires executing potentially unsafe third-party code in a host system, which raises concerns about data privacy, system stability, and unauthorized access. An attacker could inject a plug-in which looks normal from the outside, but, at runtime, might try to run malicious code. To name a few examples, it could try to encrypt the filesystem, open connections to other systems to download additional malware, or simply shut down the host.

Implementing traditional plug-in architectures in server-side applications therefore requires careful consideration of security measures to ensure the safety and integrity of the host system. The sandboxed security model of WebAssembly, together with its execution performance, makes WebAssembly a very good fit for server-side plug-in architectures.

## Server-side WebAssembly

WebAssembly is a byte code format for executable code. While initially designed for use in a web browser, it has started to conquer the server-side world as well. This expansion is driven by several key factors

**Efficiency**. WebAssembly code is designed to be fast and compact, making it an attractive option for server applications where performance is crucial.

**Portability**. Developers can package server-side logic into WebAssembly modules that are easy to distribute and run consistently across different environments. This reduces compatibility and dependency issues, which are common challenges in server deployments.

**Language interoperability**. It can be compiled from a variety of programming languages, allowing developers to use their preferred language. Going one step further, it allows to easily integrate components written in different languages by compiling them all to WebAssembly. 

**Sandboxed security**. WebAssembly's sandboxing features, which were initially designed for web browsers, also provide security benefits on the server-side. WebAssembly modules run in a secure, isolated environment, reducing the risk of vulnerabilities and exploits. This is particularly important for server applications that handle sensitive data and need to protect against potential security threats.

Server-side WebAssembly can be employed in a variety of different use cases, as presented in a [previous article](https://www.codecentric.de/wissens-hub/blog/compile-once-run-anywhere-with-wasm-and-wasi).

Here, we explore the use case of a third-party plug-in architecture. The goal of such an architecture is to allow users or customers to dynamically extend the behavior of an application. The host application provides predefined extension points. These extension points act as hooks that allow the guest WebAssembly modules to integrate seamlessly with the host application.

Users can provide their own WebAssembly modules that implement this interface. The host application invokes these guest modules at run-time, by calling the appropriate extension points.

![plugin_static.png](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/zkiaE1QTDGMcOQu5vhbw)

WebAssembly's security model guarantees safe execution of these guest modules within the host environment. The sandboxed environment isolates the guest modules, denying access to the resources of the host application. The capability-based security model allows to grant only the system calls which are needed to fulfill the task of the plug-in. By default, no system calls are possible.

## Real-world use cases

While much of the server-side WebAssembly landscape is still evolving, the plug-in architecture use case is already used in production by a few notable projects.

### Web servers and reverse proxies

The Apache HTTP Server can be extended with WebAssembly modules. This enables WebAssembly modules to respond to HTTP requests. This capability is facilitated by an extension module, <a href="https://github.com/vmware-labs/mod_wasm" target="_blank">mod_wasm</a>. 
Similarly, the Envoy proxy <a href="https://istio.io/v1.10/blog/2020/wasm-announce/" target="_blank">offers an Application Binary Interface (ABI)</a> supporting custom HTTP filters implemented as WebAssembly plug-ins. Envoy is a reverse proxy which can both be deployed as a service proxy in a service mesh architecture or as a traditional 'front proxy' between a client and the server. In both scenarios, HTTP filters intercept and/or modify the HTTP requests. This is typically done to address cross-cutting concerns such as policy checks, authentication, and telemetry data collection.

Building on this, <a href="https://www.youtube.com/watch?v=3fbgMzt6_4Q" target="_blank">a talk at the recent WASMCon 2023</a> illustrated how the investment bank Goldman Sachs is leveraging WebAssembly for their API Platform. By creating plug-ins in higher level languages such as Go or TypeScript, they are able to provide higher flexibility for their developers and increase code reuse across their various API Gateway deployments.

### SaaS providers

<a href="https://www.shopify.com" target="_blank">Shopify</a> employs WebAssembly to allow for customization of their SaaS retail platform. Shopify offers merchants to create and manage their own online stores. However, each merchant often has very specific needs and sometimes even unique business requirements, such as offering special discounts for VIP customers, selling product bundles at discounted prices, and providing customizable shipping methods. Meeting all these diverse requirements within a single software system is not practical. Therefore, extensibility is a key quality of the Shopify platform.
<a href="https://shopify.dev/docs/apps/functions" target="_blank">Shopify functions</a> allow app developers to extend the backend logic of Shopify.
Developers are required to provide a WebAssembly module containing the added logic. Additionally, they can define a GraphQL query to select the input to their plug-in. Any language that compiles to WebAssembly can be used, but Shopify provides dedicated SDKs for JavaScript and Rust to ease the development process.

### Databases

In the domain of database management systems, user-defined functions (UDFs) offer a toolkit to introduce additional logic into the existing SQL or NoSQL languages. Some database systems (<a href="https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/" target="_blank">Singlestore</a>, <a href="https://www.pingcap.com/blog/how-webassembly-powers-databases-build-a-udf-engine-with-wasm/" target="_blank">TiDB</a>) have embraced WebAssembly to define UDFs, leveraging its near-native performance while tapping into the rich ecosystems of the various source languages.
Meanwhile, the streaming data platform Redpanda <a href="https://redpanda.com/blog/wasm-architecture" target="_blank">is working towards</a> enabling WebAssembly modules as stream processors directly on their platform. In comparison to the traditional architecture of separated processors, this would move the processing logic to the storage engine, the key for low latency and high throughput.

## Example: Image-editing Web-Application

Let us have a look at an example application.

Most popular image editing software runs on the desktop environment. Typically, these applications come with a collection of image processing operations, such as special effects, denoising filters, or even advanced AI-based pipelines. Often, additional operations can be added through third-party plug-ins. Here, we conceptualize a minimal web-based image manipulation software. Users can upload images from their local filesystem, apply operations on these and receive the result. These image operations are not provided by the application itself, but are added by third-party developers. The image operation developers register their plug-ins by uploading a .wasm file through a web service. 

![plugin_runtime.png](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/0z7WvOHITwiTAO8fbvNE)

## Host-to-plug-in communication

The sandboxed nature of WebAssembly introduces challenges in the communication between the plug-in and the host. The WebAssembly instance has no access to the host system's memory. This security mechanism complicates the transfer and return of higher-level data structures such as strings, arrays, or objects. To address this challenge, several approaches have emerged.

The first common approach is to treat the guest plug-in as a command-line interface (CLI) application. Input and output are transferred through standard input and standard output, respectively. Shopify functions have <a href="https://shopify.dev/docs/apps/functions/input-output" target="_blank">employed this approach</a>, but also the WebAssembly microservice framework <a href="https://github.com/deislabs/wagi" target="_blank">WAGI</a>. It is suitable when the data is already in text form or easily representable as text.

Another approach is to shift the burden onto the host platform to copy data into and out of the plug-in's memory space. The guest plug-in must implement a method to allocate a memory block and return a pointer to its location. However, this approach is prone to memory leaks because WebAssembly lacks a garbage collector. It becomes the host's responsibility to free unused memory. This approach can only work if the initial programming language supports addressing specific memory locations.

Both these approaches are rather low-level, therefore most of the aforementioned projects provide SDKs for popular programming languages to facilitate the workflow.

## Extism

<a href="https://extism.org/" target="_blank">Extism</a> is a framework for extending applications with plug-ins written in WebAssembly. It abstracts from the low-level details described in the previous section. By offering higher-level API, it simplifies transmitting data between the host system and guest plug-in.
The project requires both the host and the guest implementations to use specific libraries - a *Host SDK* to integrate Extism in an application, and a *Plug-in Development Kit (PDK)* to write a plug-in.

Host SDKs are available for most common backend programming languages, including Python, Java, C/C++, C#, and Node. The number of PDKs is smaller, but Rust, Go, and C, the most popular programming languages targeting WebAssembly, are supported.

Here is an example on how to use the Go PDK to transfer an image as a byte array to the plug-in, apply an operation, and send the result back to the caller.
```go
import "github.com/extism/go-pdk"

//go:export applyImageOperator
func applyImageOperator() int32 {
    inputBytes := pdk.Input()

    result := applyOperatorOnJpeg(inputBytes)

    mem := pdk.AllocateBytes(result.Bytes())
    pdk.OutputMemory(mem)

    return 0
}
```
The exported function's signature does not expose any parameters. Instead, the input byte data can be retrieved `pdk.Input()`. Returning data is done by first allocating a chunk of memory, then returning the content with `pdk.OutputMemory(mem)`. 

The following code shows how the same plug-in can be called from a Python host.
```python
from extism import Plugin

def apply_operator(wasm_file, image_bytes):
    with open(wasm_file, "rb") as f:
        wasm_bytes = f.read()
    config = {"wasm": [{"data": wasm_bytes}]}
    plugin = Plugin(config, wasi=True)
    plugin.call("applyImageOperator", image_bytes)
```
The plug-in can be invoked by referencing the previously implemented method and passing the image as a parameter. The process of manually copying byte arrays into and out of the WebAssembly instance is abstracted away.

The two blocks of code were minimized for simplicity. A real-world application could include logging and error handling, restrict the memory of the plug-in, or validate its signature, all of which can be covered with the Extism SDKs.

The complete implementation of this project is available <a href="https://github.com/j-blue-arz/image-processing-wasm" target="_blank">here</a>.

## Security considerations

WebAssembly was initially created for running within web browsers. Therefore it was designed with security in mind.

Capability-based security is a fundamental aspect of WebAssembly's design. It ensures that access to system resources and capabilities has to explicitly be granted by the host VM. Extism (like all other WebAssembly runtimes) has all system calls deactivated by default. If, in the example code above, a plug-in tried to call a remote API, the runtime would raise an exception. Extism allows granular access control, enabling HTTP and file system access on a per-host or per-file/directory basis. Additionally, more general capabilities can be enabled by injecting host functions into the plug-in.

WebAssembly also incorporates a form of call stack protection. The memory a WebAssembly application has access to is separated from the call stack, and also from the memory containing its own code. This aims to protect against stack-based buffer overflow attacks such as the ones described in a <a href="https://arxiv.org/abs/2111.01421" target="_blank">2021 paper</a>.

However, these security mechanisms must be implemented both in the WebAssembly VM (host runtime) and in the language-specific compiler toolchain. This brings some risk, due to the increasing number of runtimes and supported languages. A <a href="https://www.usenix.org/conference/usenixsecurity20/presentation/lehmann" target="_blank">paper from 2020</a> examines the security of emscripten, the C++-to-WebAssembly compiler toolchain. At the time, they came to the conclusion that many compiler security features for native compilation targets were not active when compiling to WebAssembly. The paper highlights the need for improvements both in the compilers and in the runtimes to ensure the security of WebAssembly applications.

Furthermore, it's important to note that while WebAssembly itself can protect the host system, the output produced by the plug-ins should still be treated with caution. This is not an issue of WebAssembly - in any plug-in architecture, there is a risk that the output generated by a plug-in could be malicious.
In the image-editing application described above, an attacker could build a plug-in that hides executable code within images. Or they could generate completely different images with inappropriate or forbidden content. The output of a third-party plug-in should be handled with the same care as user-generated data.

## Alternatives

WebAssembly is not the only option for enabling plug-in architectures. There are other alternatives worth considering.

One alternative is the use of native language extensions or dynamically linked libraries. These options offer high performance due to their native code, but they lack isolation as they share memory and address space with the host application. Additionally, they have limited portability as they need to be compiled specifically for the target environment.

Another alternative is the use of scripting languages like Python, Lua, or JavaScript, which can be run embedded within a host application. These dynamically typed languages can provide reasonable performance for small scripts, particularly when a runtime with a Just-In-Time (JIT) compiler is utilized. However, they may not be suitable for larger performance-critical applications due to the nature of the JIT and garbage collection. In addition, developers may need to learn a new language, which can be a potential drawback. In contrast, WebAssembly can be targeted by multiple languages, offering more flexibility in choosing a familiar programming language.

A common approach for extending behavior in SaaS environments is through HTTP calls to external APIs. This involves registering a URL, e.g. as a webhook to be called when needed. While this approach guarantees the safety and security of the host application, it comes with the downside of high latency. Making remote API calls introduces delays due to network communication, which may not be suitable for applications requiring real-time interactions.

## Java

Compiling Java to WebAssembly to run in the browser and interacting with JavaScript is facilitated by open source projects such as <a href="https://teavm.org/" target="_blank">TeaVM</a> and <a href="https://cheerpj.com/" target="_blank">CheerpJ</a>. However, the support for targeting the WebAssembly System Interface (WASI) in the Java ecosystem is currently limited. Notably, Extism does not provide a plug-in-SDK for Java.

While there is a <a href="https://github.com/fermyon/teavm-wasi" target="_blank">TeaVM fork</a> which aims at compiling Java to server-side-compliant WebAssembly, the support for building WebAssembly plug-ins in Java lags behind compared to other languages. For one, it is challenging to transfer data into the plug-in, due to the memory model of Java. Java lacks the ability to work with raw addresses, and the memory location of objects on the heap can change with each Garbage Collection (GC) cycle. The situation might improve with better support once <a href="https://github.com/WebAssembly/gc" target="_blank">native GC</a> is built into WebAssembly. Additionally, certain parts of the Java standard library rely on native system calls that are not covered by WASI. An example of this is encoding and decoding image files.

## Conclusion

Plug-in architectures offer crucial flexibility in software systems. WebAssembly presents a new and promising approach for implementing them on the server side. While still a young technology, WebAssembly plug-ins are already being used in production projects.
Exploring the potential of WebAssembly for efficient and secure server-side plug-ins is an exciting opportunity in software development.

Plug-in architectures with WebAssembly

Market Lead Energy & Utilities

Senior Online Marketing Manager

Leiter Region Südwest

<p>Digitalisierung revolutioniert jedes Business und das schon seit über einer Dekade. Dieser andauernde Trend wird auch Ihr Business-Modell nicht unberührt lassen und hat einiges zu bieten. Es gibt zahlreiche Beispiele, wie und wo eine digitale Transformation helfen kann. – Ein neues Kundenportal, um die herausfordernden Anforderungen des Kunden zu bedienen, oder gleich komplett neue Zielgruppen anzusprechen. Alternativ helfen innovative Software-Lösungen beim Ersetzen von manuellen und zeitkonsumierenden, oftmals auch fehleranfälligen Prozessen. Sie helfen auch beim Erzeugen einer modernen Datenanalyse-Plattform, um “state of the art” künstliche Intelligenz auf die wertvolle Ressource Daten anzuwenden. Die Möglichkeiten sind unendlich und vielfältig. Aber so einfach es zunächst klingen mag, sehr oft gibt es Abhängigkeiten in der Organisation oder anderen Bereichen, die es einfach nicht ermöglichen ein neues Experiment zu starten. Besonders für die kleinen und mittleren Unternehmen ist es oft riskant, wenn nicht sogar unmöglich, ein neues Entwicklungsteam mit den benötigten Erfahrungen auf die Beine zu stellen. Wenn doch, dann bedarf es vieler Zyklen an Planung, Entwicklung, Testen und Messen bevor überhaupt ein Kunde die Anwendung zu Gesicht bekommt.</p><p>Die so entstehende Verzögerung des echten Kundenfeedbacks führt mit einer hohen Wahrscheinlichkeit dazu, dass das Produkt fehlschlägt. Denn nur zeitnahes Feedback verhindert Missverständnisse des zu lösenden Problems oder Fehlinterpretationen der Anforderungen.</p><h2><span>Entwicklung einer kundenzentrischen Software, schnell und verlässlich</span></h2><p><span>Solution Factory ist ein kundenzentrierter, schneller und verlässlicher Ansatz, um neue Ideen zu verproben und innovative zukunftsweisende Lösungen durchzuführen. Die Idee und die User Experience stehen im Mittelpunkt dieses Vorgehens. Sie sind die maßgeblichen Faktoren anhand derer der Erfolg des Projekts evaluiert wird. In enger Zusammenarbeit mit den Domain-Experten und dem Endbenutzer wird der Kunde durch die gesamte Reise der Problemeingrenzung geführt – durch das Bauen eines Minimal Viable Product (MVP), das sich bis hin zu einem vollständigen Produkt entwickeln läßt. Solution Factory setzt sich aus drei Phasen zusammen: Design Sprint, MVP Development und Solution Development. </span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qJLiBnYlSHiZFS8vEH8d"><img width="250" height="175" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qJLiBnYlSHiZFS8vEH8d" alt /></a> </p><h2><span>Design Sprint</span></h2><p><span>Das Konzept des Design Sprints basiert auf der Lean-Startup-Methodik. Diese erlaubt es uns, disruptive Geschäftsmodelle zu designen während gleichzeitig die Risiken berücksichtigt werden. Wir definieren eine neue Geschäftshypothese (</span><i><span>idea</span></i><span>) und übersetzen (</span><i><span>build</span></i><span>) diese in Code. Dann wird diese Hypothese gemessen (</span><i><span>measure</span></i><span>) und die Ergebnisse analysiert (</span><i><span>data</span></i><span>). Daraus entwickeln sich neue Ideen, die wir im Nachgang optimieren (</span><i><span>learn</span></i><span>). Hierdurch ist es möglich, den Benutzer besser zu verstehen und ihn somit in den Prozess für neue Hypothesen zu integrieren. Daraus bildet sich eine Beweiskette von Möglichkeiten die funktionieren, und Möglichkeiten die nicht funktionieren. Durch iteratives Vorgehen können neue Ideen schnell getestet und schlechte Ideen schneller verworfen werden, ohne den Verlust von Endkunden oder Budget. </span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/W5l8MenT4eUOq2GcaWdL"><img loading="lazy" width="250" height="249" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/W5l8MenT4eUOq2GcaWdL" alt /></a> </p><p><span>Mit neuen Produkten hat man aber ein sogenanntes “Henne-Ei-Problem”, oder auch “Kausalitäts-Dilemma”: Ohne eine “theoretische” Lösung können wir keine Daten sammeln und ohne Daten keine neue Hypothese aufstellen. Der Design-Sprint-Ansatz versucht hier einen Kurzschluss. Endkunde, Domain-Experte und Stakeholder werden zusammen gebracht, um gemeinsam ein klares Bild des Problems zu definieren. Auf dieser Basis wird der Prototyp der Lösung skizziert, um direkt vom Endkunden zu lernen, bevor auch nur eine einzige Zeile Code geschrieben wurde. </span></p><p><span>Um es in der Sprache des “Lean-Software-Development” auszudrücken: Es wird direkt aus der Geschäftshypothese gelernt, ohne viele Daten zu sammeln.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/O42yfljFTOyuW8IjQ0Fs"><img loading="lazy" width="250" height="249" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/O42yfljFTOyuW8IjQ0Fs" alt /></a> </p><h2><span>MVP Development</span></h2><p><span>Die MVP-Entwicklung startet direkt nach dem Design Sprint. Sie besteht aus sechs Software-Entwicklungs-Sprints, ein Sprint pro Woche. Jeder Sprint startet montags und endet freitags mit einer Sprint Review, bei der die Sprint-Ergebnisse (Features) gezeigt werden und der nächste Sprint geplant werden kann. Nach sechs Sprints ist diese Phase beendet, und das Produkt, ein MVP, wird an den Kunden übergeben. Das MVP ist eine funktionierende Anwendung, welche die Schlüsselaspekte der Ziellösung bereits abbildet und durch den Endkunden genutzt werden kann. Es folgen zwei Wochen Entwicklungspause während der das MVP durch den Benutzer getestet werden kann. </span></p><p><span>In dieser Phase stehen alle Entscheidungswege offen. Der Kunde kann an dieser Stelle entscheiden, ob aus dem MVP ein Produkt entwickelt wird, oder ob die Hypothesen nochmal überarbeitet werden müssen.</span></p><h2><span>Solution Development</span></h2><p><span>Die Solution-Development-Phase schließt an das MVP an. Hier wird aus dem MVP ein Produkt gebaut, das letztendlich so auch in Produktion läuft. </span></p><p><span>In dieser Phase ist es das Hauptaugenmerk, festzustellen, welche Funktionalitäten im MVP fehlen, um es zu dem gewünschten Ziel zu bringen. Noch während der Entwicklung des MVPs war das Ziel, dem Endkunden ein minimales Set an Inhalten zum Experimentieren zur Verfügung zu stellen. Um dieses Ziel zu erreichen, ist es notwendig, ein gesundes Maß an technischen Schulden aufzubauen. Während der Solution-Development-Phase werden diese technischen Schulden iterativ abgebaut, so das aus dem MVP eine produktionsfähige Lösung entsteht.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/XTM1vqUIQcujZhCLZkrE"><img loading="lazy" width="250" height="175" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/XTM1vqUIQcujZhCLZkrE" alt /></a> </p><h2><span>Success Story</span></h2><p><span>Unserer Erfahrung nach ist Solution Factory eines der effektivsten Modelle, um neue Projekte und Produkte anzustoßen. Experimentieren mit neuen Ideen – mit einem Minimum an Risiko und einem Maximum an Kundenzufriedenheit. Im Folgenden erläutern wir beispielhaft, wie Solution Factory geholfen hat, ein immanentes Problem im internen Kunden-Onboarding-Prozess unseres Kunden zu lösen.</span></p><h2><span>Das Problem</span></h2><p><span>Unser Kunde, einer der größten in seinem Sektor in Deutschland, ist gesetzlich dazu verpflichtet, einen komplexen Onboarding-Prozess für seine neuen Kunden zu durchlaufen: Zahlreiche Informationen müssen gesammelt, unterschiedliche rechtliche Aspekte beachtet und Verträge unterzeichnet werden, und das zwischen unterschiedlichen Dienstleistern. Der Prozess verteilt sich über viele teilnehmenden Partner und benötigt teilweise Wochen, bevor er abgeschlossen werden kann. </span></p><p><span>Um all diese Komplikationen zu bewältigen, waren viele manuelle Schritte im direkten Kontakt mit den Kunden notwendig. Ein Mitarbeiter musste dabei kontinuierlich betreuen, alle Schritte mit durchlaufen, den andauernden Prozessablauf sicherstellen und die Kunden an die Berichtigung von fehlenden Informationen oder an nahende Fristen erinnern. Auf der einen Seite erhöhte dieser manuelle Prozess die Aufwände bei unserem Kunden erheblich, zum anderen verschlechterte sich aufgrund dieses Prozesses die Kommunikation mit dem Endkunden und führte oft zu Verwirrungen, erhöhte die Fehlerrate, verzögerte den Prozess und schadete so der Kundenzufriedenheit.</span></p><h2><span>Die Lösung</span></h2><p><span>Zuerst wurde eine Gruppe aus Design Sprint Coaches, einem User-Experience-Experten, einem User Interface (UI) Designer und einem Software- und Cloud-Architekten gegründet. Seitens des Kunden wurden sowohl ein technischer Experte als auch ein repräsentativer Endbenutzer in das Projekt involviert.  Alle Beteiligten trafen sich für den Design Sprint in unserem Office in Karlsruhe getroffen.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/XCdZwjbYTRCMABAqCfTD"><img loading="lazy" width="250" height="175" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/XCdZwjbYTRCMABAqCfTD" alt /></a> </p><p><span>Der Montag diente dem Ziel, die Kundendomäne und das eigentliche Problem zu verstehen. Die Domänenexperten durften die aktuelle Situation beschreiben und ihre Sicht auf die aktuell größten Schwierigkeiten darstellen. Das gesamte Team partizipierte aktiv bei dieser Diskussion, indem Fragen gestellt und weitere Punkte gesammelt wurden. Das Ergebnis wurde anschließend gruppiert und formalisiert in eine eindeutige Beschreibung des Problems gefasst. Dieses definierte das Ziel des Design Sprints. Die Zusammenarbeit am Montag zeigte sich als sehr wertvoll für das gesamte Team, da es den Scope des Design Sprints klar definierte und half, eine gemeinsame Sprache zwischen Domänen- und IT-Experten zu finden. </span></p><p><span>Nachdem das Problem identifiziert war, wurde der Dienstag der Lösungsfindung gewidmet. Jeder Teilnehmer wurde nach seinen Ideen befragt und sollte skizzieren, wie eine mögliche Lösung aussehen könnte. Am Ende des Tages wurden die Skizzen dem Team präsentiert, um potentielle Diskussionen in den nächsten Tag zu schieben. Am Mittwoch und Donnerstag wurden die Skizzen evaluiert und die Ideen, die dahinter standen, in einem finalen Lösungsansatz zusammengefasst. Diese Idee wurde dann in einen “Click-Dummy-Prototypen” umgesetzt, welcher die komplette User Journey ohne wirkliche Implementierung nachahmte. </span></p><p><span>Der Freitag war der letzte und der aufregendste Tag dieser intensiven Woche aus Know-How-Transfer, Diskussionen und Ideengenerierung. Obwohl wir jetzt eine Vorstellung hatten, wie das Produkt aussehen kann, gab es keine Garantie, dass es den echten Anforderungen des Endkunden gerecht werden würde. Es war immer noch unklar, ob das eigentliche Problem damit gelöst wird. Daher wurden am Freitag fünf separate Interviews mit der Zielgruppe des Produktes organisiert. Diese waren sich der Ausgangssituation durchaus bewusst und durften die Lösung zum ersten mal sehen. Jedes Interview dauerte zwischen 45 und 60 Minuten. Am Ende wurde der Interviewte gebeten, das Erlebte in Summe zu bewerten. </span></p><p><span>Auf einer Skala von 1 bis 10 erreichte der Prototyp einen Durchschnitt von 8,5. Interessanterweise fragten alle Interviewten, wann denn das Produkt zur Verfügung stehen würde. Dies war eindeutig ein Zeichen des Erfolgs und gab grünes Licht für die nächste Phase.</span></p><h2><span>Resultat</span></h2><p><span>Ein Kunden-Management-Portal ist konzipiert. Es bildet den kompletten Onboarding-Prozess für neue Kunden ab und ermöglicht es den Kunden, ihre Produkte jederzeit selber zu pflegen, ohne irgendeine weitere manuelle Interaktion. Der Benutzer kann neue Verträge anlegen, bereits existierende im Prozessverlauf verfolgen und die nächsten Schritte überwachen. Das Portal verfolgt alle Verträge und führt den Kunden durch alle notwendigen Schritte. Automatische Erinnerungen stellen sicher, dass keine Termine verpasst oder übersehen werden. </span></p><p><span>Während der folgenden sechs Wochen nach dem Design Sprint wurde das MVP in den Räumen der codecentric AG in enger Zusammenarbeit mit den Domänen-Experten des Kunden entwickelt. Das MVP wurde anschließend evaluiert und ist in den folgenden 12 Monaten bis hin zur endgültigen Lösung weiterentwickelt worden. </span></p><p><span>Das Entwicklungsteam bestand aus einem Architekten aus Karlsruhe zusammen mit zwei Full-Stack-Entwicklern aus dem codecentric Büro in Banja Luka (Bosnien) und einem Designer vom codecentric-Partner UX&I aus Düsseldorf. </span></p><p><span>Für ein optimales Team Building waren die Kollegen aus Bosnien in den ersten drei Wochen der MVP-Entwicklung in Karlsruhe und anschließend remote am Projekt beteiligt. Sie waren während des Projektes in konstantem Kontakt und  durch Ad-hoc-Kommunikation und reguläre Meetings immer mit dem Rest des Teams synchronisiert. Trotz der Verteilung im Team wurde eine hohe Geschwindigkeit und Effizienz erreicht. Dank der Cloud-Native- und Serverless-Architektur mit einer Continuous Delivery Pipeline war es möglich, die Stakeholder eng in den Entwicklungsprozess zu involvieren und kurze Feedback-Zyklen zu etablieren. Auch die Infrastrukturkosten wurden dank dieses Vorgehens deutlich reduziert.</span></p><h2><span>Fazit</span></h2><p><span>Klassische Software-Entwicklung ist oft langsam, im Sinne von Kundenfeedback und Adaption an das Feedback. Das erhöht das Risiko und die Kosten von konventionellen Vorgehen deutlich, wenn mit neuen Ideen und Hypothesen experimentiert werden soll. </span></p><p><span>Solution Factory offeriert hier einen neuen Ansatz: die Entwicklung von Kunden-zentrischen Software-Produkten. </span><span><br /> </span><span>Es gibt disruptiven Innovationen in einer Organisation Platz sich zu entfalten, bei Minimierung des Risikos, der Kosten und einer erhöhten Feedbackschleife. <a href="/standorte/karlsruhe" target="_blank"><strong>Wenn Sie mehr über die Konzepte der Solution Factory wissen wollen und wie es Ihnen helfen kann, neue Hypothesen zu validieren, melden Sie sich bei uns.</strong></a> </span></p>

Solution Factory – In 9 Wochen von der Idee zum Produkt

IoT-Analyse-Plattform

<p>Getting the best of Apache Cassandra inside Apache Karaf: this blog post will describe how easy it was to embed the NoSQL database inside the runtime. This can be helpful while developing OSGi-related applications with Karaf that work together with Cassandra.<br /> The entire project for the examples can be found at <a href="https://github.com/ANierbeck/Karaf-Cassandra">GitHub</a> .</p><p>Working with Apache Cassandra is fun, though sometimes you just need a simple embedded server with a connected client to test something for a POC or to explore the APIs. So it should be easy to install and use this embedded server. Combining this with the power of Apache Karaf, an OSGi Container, makes an unbeatable team.</p><p><span></span></p><h2>Apache Karaf</h2><p>Apache Karaf is an application server for OSGi applications. In the showcase we are going to use Karaf in the version 4.0.0-SNAPSHOT. It is still unreleased, but comes with so many great improvements, that we chose the SNAPSHOT anyway.</p><p>The main goal of Apache Karaf is to provide a complete infrastructure to run OSGi related applications without the hassle of integrating and fine-tuning the components. For example it provides standard logging functionality, a shell, which can also be called via SSH and some more infrastructural bundles. But certainly one of the biggest plusses is the shell that even supports auto completion and acts like a regular Unix shell.</p><h2>Apache Cassandra Embedded</h2><p>Running an embedded Apache Cassandra isn’t that hard, you just need to instantiate a <em>org.apache.cassandra.service.CassandraDaemon</em>. Without a configuration the daemon would fail to start though. Providing such a configuration with Karaf or OSGi in general is quite easy. Implement a <a href="http://www.osgi.org/javadoc/r4v42/org/osgi/service/cm/ManagedService.html">ManagedService</a>  that manages the lifecycle, i.e.. start, stop, configure, of the wrapped CassandraDeamon .</p><p>As a ManagedService the newly implemented OsgiEmbeddedCassandra class just needs to implement the update method of the ManagedService API and therefore will be informed if a new configuration for the embedded Cassandra service is available. This method will take care of starting and stopping the CassandraDeamon.</p><p>The availability of a new configuration is triggered by the registered persistentID <strong>de.nierbeck.cassandra.embedded</strong>, if there is a new configuration available the new configuration for the cassandra.yaml file is stored in the service and the daemon will be stopped and started again.</p><div>

```java5
public class OsgiEmbeddedCassandra implements Server, CassandraService,
		ManagedService {
...
	@Override
	public void updated(Dictionary<String, ?> properties)
			throws ConfigurationException {
		if (isRunning())
			stop();
		if (properties != null) {
			cassandraConfig = (String) properties.get("cassandra.yaml");
		}
		start();
	}
...
 
}
```

</div><p>Additionally the service will implement the methods to start and stop the embedded Cassandra via the implemented interface.</p><div>

```java5
...
public class OsgiEmbeddedCassandra implements Server, CassandraService,
		ManagedService {
...
	@Override
	public void start() {
		logger.info("starting Cassandra in Embedded mode");
 
		if (cassandraConfig != null) {
			System.setProperty("cassandra.config", "file://" + cassandraConfig);
		}
		System.setProperty("cassandra-foreground", "false");
 
		cassandraDaemon = new CassandraDaemon();
		try {
			logger.info("initializing cassandra deamon");
			cassandraDaemon.init(null);
		} catch (IOException e) {
			throw new RuntimeException(e);
		}
		logger.info("starting cassandra deamon");
		cassandraDaemon.start();
 
		logger.info("cassandra up and runnign");
	}
 
	@Override
	public void stop() {
		logger.info("Stopping cassandra deamon");
		logger.info("cleaning up the Schema keys");
		Schema.instance.clear();
		logger.info("stopping cassandra");
		cassandraDaemon.stop();
		logger.info("destroying the cassandra deamon");
		cassandraDaemon.destroy();
		logger.info("cassandra is removed");
		cassandraDaemon = null;
 
		logger.info("removing MBean");
		MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
		try {
			mbs.unregisterMBean(new ObjectName(
					"org.apache.cassandra.db:type=DynamicEndpointSnitch"));
		} catch (MBeanRegistrationException | InstanceNotFoundException
				| MalformedObjectNameException e) {
			logger.warn("Couldn't remove MBean");
		}
 
	}
...
}
```

</div><p> </p><h2>Creating Karaf – Cassandra administration commands</h2><p>The interactive Karaf command shell is a powerful tool, which helps a lot with every-day debugging and administration tasks. So it is just natural to have some administrative commands available to control the embedded Cassandra service.<br /> In this showcase we’ll create four Commands to manage the embedded Cassandra service. The <em>StopService</em> command will, as the name already states, give you the ability to stop a running embedded Cassandra service. The <em>StartService</em> command starts the service, if it isn’t already running. With the <em>IsServiceRunning</em> command you’ll get feedback about the current state. As an embedded instance is usually good for starting a POC or some showcase you might want to do some more try and error, so a cleanup of the mess could be handy. For this the <em>CleanupKeyspace</em> command is used. How easy it is to create such a command can be seen by the following StartService command, this and all other commands can be found in the sources.</p><div>

```java5
@Command(scope = "cassandra-admin", name = "start", description = "Connect to cassandra server")
@Service
public class StartService implements Action {
 
	@Reference
	CassandraService cassandraServer;
 
	@Override
	public Object execute() throws Exception {
		if (cassandraServer.isRunning()) {
			System.err.println("Embedded Cassandra is already started");
			return null;
		}
 
		cassandraServer.start();
		System.out.println("Embedded Cassandra started.");
		return null;
	}
 
}
```

</div><p>It’s a very simple class, all of the infrastructure needed is hidden by annotations. The <em>@Command</em> annotation marks this service as a command. The properties tell the scope of the command on the shell, like <strong>cassandra:connect</strong>. This is useful to group certain commands.<br /> The <em>@Service</em> annotation declares this class to be a service. With this, it will be registered as a service in the service registry of the OSGi framework. The embedded Cassandra service is referenced by the <em>@Reference</em> annotation. Karaf will take care of injecting the service if it is available, otherwise this command won’t be accessible from the shell. As this command doesn’t use any further arguments only the execution method is needed. It’ll start the service if the underlying CassandraDeamon is stopped, otherwise it fails with a message. The execute method which is called by the shell implementation, does expect some return value, which will be printed to the shell. But since we want to be in control of the output, especially in case of an error, the method just returns a null.</p><h2>Installation in Karaf</h2><p>Karaf uses a concept called Features to simplify the grouping and deployment of bundle onto the server. A Karaf feature describes a set of bundles to be installed in one go. The feature descriptor is a xml structure.<br /> To install the bundles required for running an embedded Cassandra interacting with it via the shell, you’ll just need to create a feature descriptor. The following snippet shows a fragment of the Karaf feature to install the showcase:</p><div>

```xml
<features xmlns="http://karaf.apache.org/xmlns/features/v1.3.0" name="Karaf-Cassandra-Feature-1.0.0-SNAPSHOT">
...
    <feature name="Karaf-Cassandra-Embedded" description="Karaf-Cassandra-Feature" version="1.0.0.SNAPSHOT">
        <bundle>mvn:de.nierbeck.cassandra/Karaf-Cassandra-Service/1.0.0-SNAPSHOT</bundle>
        <bundle>mvn:de.nierbeck.cassandra/Karaf-Cassandra-Embedded/1.0.0-SNAPSHOT</bundle>
        <bundle>mvn:de.nierbeck.cassandra/Karaf-Cassandra-Admin/1.0.0-SNAPSHOT</bundle>
    </feature>
...
</features>
```

</div><p>To install this feature you’ll need to go to the shell of Karaf and issue the following commands. Where the first command registers the feature definition and the second one does install the feature.</p><div>

```bash
feature:repo-add mvn:de.nierbeck.cassandra/Karaf-Cassandra-Feature/1.0.0-SNAPSHOT/xml/features
feature:install Karaf-Cassandra-Embedded
```

</div><p>After this feature is installed, you are able to use the commands we previously defined. For Example issuing the following command:</p><div>

```bash
cassandra-admin:isRunning
```

</div><p>Will return true, as the embedded Cassandra is running.</p><div><a href="http://notizblog.nierbeck.de/wp-content/uploads/2014/12/Karaf-Cassandra-Feature-Install.mp4">http://notizblog.nierbeck.de/wp-content/uploads/2014/12/Karaf-Cassandra-Feature-Install.mp4</a> </div><h2>Some more shell commands</h2><p>As it is pretty boring just to start and stop an embedded Cassandra server, we will add some more commands to our library. Those commands don’t necessarily need an embedded Cassandra server they could be used to connect to any Cassandra cluster.<br /> At first some Cassandra client bundles need to be installed. In the previously mentioned feature definition is also a feature defined for Cassandra client bundles. Just install the Karaf-Cassandra-Client feature as shown before. Now we need some more client command, such as connecting to a Cassandra Cluster or issuing CQL scripts. The following commands are available if you install the last remaining feature in the feature descriptor, the Karaf-Cassandra-Shell feature.</p><ul><li><strong>cassandra:connect:</strong><br /> This command does connect to any known cluster just issue the name or IP address of the node known to run a Cassandra instance. If the port isn’t the default one use the –p option to add an alternative port. To connect to the embedded Cassandra server issue the following:<br /> <em>cassandra:connect –p 9142 localhost</em></li><li><strong>cassandra:disconnect:</strong><br /> Disconnects the current shell session from the remote Cassandra cluster. In case there is no active session it’ll tell you.</li><li><strong>cassandra:isConnected:</strong><br /> Will tell you true or false, if there is a connection bound to the shell. In case a USE command for a certain keyspace has been issued, the isConnected will return true:keyspace_name</li><li><strong>cassandra:cql:</strong><br /> This command takes an argument or an option. In case you give an argument like the following:<br /> <em>cassandra:cql “select * from foo.bar;”</em><br /> it will print a table with the selection of table bar in keyspace foo.<br /> If you call:<br /> <em>cassandra:cql –f /absolut/path/to/select.cql</em><br /> it will parse the file and execute the contained cql script. Printing tables in case there are Rows to print.</li></ul><p>These are some nice commands but sometimes when debugging for an issue you don’t really know where to start, so a little help by the system is needed. For this some extra cassandra:cqlsh subshell commands are available. The cqlsh subshell is intentional because the following commands are supposed to give the same feeling as in the cqlsh, which is a python script. Those commands have completers that take the current scope like session, selected keyspace or table into account.<br /> To use those commands, either switch to the corresponding subshells by issuing cassandra and cqlsh or prefix the commands by cassandra:cqlsh:</p><ul><li><strong>USE:</strong><br /> The USE command appended by the keyspace name will use this keyspace name for the current Cassandra session, which is bound to the shell. This command does have a completer which tells you of the available keyspace names.</li><li><strong>DESCRIBE:</strong><br /> This command issued with keyspaces will tell you about the available keyspaces. If used with TABLES, it’ll tell you of the known tables and if it is combined with TABLE table_name, it’ll tell you about the details of the selected table. Again this command does have a completer to help about the keyspaces and tables.</li><li><strong>DROP:</strong><br /> The DROP command will drop either a table or a keyspace, depending on the input. This command also provides a completer for navigating to the right command.</li><li><strong>CREATE:</strong><br /> Supports creating either a keyspace or a table structure, together with the completer it should provide a CQLSH like completion and handling.</li><li><strong>INSERT:</strong><br /> This command will insert new data into a select table. A completer helps on finding the right syntax for it.</li><li><strong>SELECT:</strong><br /> The SELECT command will be send to the cluster, if there is a valid result it will be printed as a table view. The building of the select term is supported by the completion to help like in the CQLSH.</li></ul><p>The most power of this showcase comes from the CQLSH like commands, which help debugging into the data contained in your Cassandra Database. In combination with the embedded Cassandra both tools provide a nice combination of features to start and play with a Cassandra Database for a new Project or POC. Karaf itself is just there to start and help with the infrastructure, as a lightweight container.</p><div><a href="http://notizblog.nierbeck.de/wp-content/uploads/2014/12/Karaf-Cassandra-ShellWithCompletion.mp4">http://notizblog.nierbeck.de/wp-content/uploads/2014/12/Karaf-Cassandra-ShellWithCompletion.mp4</a> </div><p>To run all this yourself follow the description of the Showcase at</p><p><a href="https://github.com/ANierbeck/Karaf-Cassandra/">https://github.com/ANierbeck/Karaf-Cassandra/</a> </p><p> </p>

Combining Apache Cassandra with Apache Karaf

<p>In the world of Internet of things (IoT) you work with a continuous flow of data. For this you have two options at hand, the first is to do batch processing long after the data is collected. The other option is to analyse the data while it is being collected. How to do this Fast Data approach can be found in this blog post: <a href="/wissens-hub/blog/iot-analytics-platform" target="_blank">IoT Analytics Platform</a> <br /> <br /> The focus of the previous article had been on the SACK part of the SMACK Stack. That would be Apache Spark, Akka, Apache Cassandra and Apache Kafka. The following article will show how to complete the picture of a SMACK Stack application and how to actually run it with DC/OS and Mesos, for which the M in SMACK stands.</p><h2>Infrastructure</h2><p>So what’s the infrastructural part which is needed to run this fine SMACK application?<br /> First of all, Mesos and the corresponding modules like marathon, chronos and DC/OS are needed. As Mesos itself is limited and only gets its kicks from working with DC/OS, we’ll focus on DC/OS. As the description of how to best <a href="https://github.com/zutherb/terraform-dcos" target="_blank">install DC/OS with terraform</a>  is a topic of its own. We’ll skip that and depend on a pre-configured environment.<br /> Now that the underlying “operating system” is available, let us take a closer look at what else is needed. As the S in SMACK stands for Spark, we will need to install the Spark Framework in DC/OS.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Z9uEZGaLTLCnPp0B2kuh"><img width="891" height="500" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Z9uEZGaLTLCnPp0B2kuh" alt="dcos_install_spark" /></a> </p><p>This Spark framework starts and keeps track of the Apache Spark master server, which will take care of our soon to come Spark jobs.<br /> So now we’re set with Apache Spark and ready to actually digest the incoming data, but to do this, we need some data to work with. Therefore we will need an Apache Kafka Server first, that’s another easy task that can be installed in the same way as we’ve done previously.<br /> Now that the consuming side is satisfied, we need a way to get our data into Kafka and since we’re able to consume via the Spark job, but the job has no place to put the data to. Therefore another part of the SMACK stack is needed to be installed: Apache Cassandra.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mEXTCmqSEqBBYFw7wTNc"><img loading="lazy" width="430" height="445" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mEXTCmqSEqBBYFw7wTNc" alt="dcos_install_cassandra" /></a> </p><p>After that is done, Cassandra itself doesn’t know how to store the data, therefore it’s crucial to initialize the database with a schema. Chronos is there to the rescue, we’re able to add add a job. But since the Job needs a Docker image, we can’t do that via the Web UI, we’re gonna use the REST-API instead.<br /> To add the new Job we need to issue the following curl command:</p><div>

```bash
bash curl -L -H 'Content-Type: application/json' -X POST -d '{json hash}' master.mesos/service/chronos/scheduler/iso8601
```

</div><p>The <em>{json hash}</em> needs to be replaced by the following json syntax, which can be used for creating this job. Just make sure that the placeholder for cassandra host is replaced by the actual dns name entry.</p>

```

{
    "schedule": "R0/2014-03-08T20:00:00.000Z/PT1M",
    "name": "init_cassandra_schema_job",
    "container": {
        "type": "DOCKER",
        "image": "codecentric/bus-demo-schema",
        "network": "BRIDGE",
        "forcePullImage": true
    },
    "cpus": "0.5",
    "mem": "512",
    "command": "/opt/bus-demo/import_data.sh $CASSANDRA_HOST",
    "uris": []
}

```

<p>This will generate a new job with chronos, as it can be seen in the chronos overview:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GhLJPAKZQgmERRJk9Jqp"><img loading="lazy" width="1999" height="1015" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GhLJPAKZQgmERRJk9Jqp" alt="dcos_chronos_overview" /></a> </p><p>Now make sure that this job is executed at least once to create the cassandra keyspaces needed for the application.</p><h2>Easiness of Installation</h2><p>Now that the foundation for our application is running we need to install our own application. The digesting part with consuming from Kafka and publishing to Cassandra is possible. Now how do we get our own Docker images containing of the Akka Ingest part into our cluster? That is quite easy the installation of an application contained inside a docker image can be achieved by using marathon. Let’s start with a new Application:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/VjGCg36LTsSA4xFUjRxA"><img loading="lazy" width="741" height="680" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/VjGCg36LTsSA4xFUjRxA" alt="dcos_install_custom_json" /></a> </p><p>In this application we switch to the JSON mode and add a JSON snippet to it, as can be seen below:</p>

```

{
  "id": "/ingest",
  "cpus": 1,
  "mem": 2048,
  "disk": 0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "codecentric/bus-demo-ingest",
      "network": "HOST",
      "privileged": false,
      "parameters": [],
      "forcePullImage": true
    }
  },
  "env": {
    "CASSANDRA_HOST": "$CASSANDRA_HOST",
    "KAFKA_HOST": "$KAFKA_HOST",
    "KAFKA_PORT": "$KAFKA_PORT"
  }
}

```

<p>But beware, we need to change those settings for cassandra and kafka host and port to the actual names used in our cluster. The same needs to be done for the Akka Http Service Docker image which also contains the front-end part.</p>

```

{
    "id": "/dashboard",
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "codecentric/bus-demo-dashboard",
            "network": "HOST",
            "forcePullImage": true
        }
    },
    "acceptedResourceRoles": [
        "slave_public"
    ],
    "env": {
        "CASSANDRA_HOST": "$CASSANDRA_HOST",
        "CASSANDRA_PORT": "9042"
        "KAFKA_HOST": "$KAFKA_HOST",
        "KAFA_PORT" : "9092"
    },
    "healthChecks": [
        {
          "path": "/",
          "protocol": "HTTP",
          "gracePeriodSeconds": 300,
          "intervalSeconds": 60,
          "timeoutSeconds": 20,
          "maxConsecutiveFailures": 3,
          "ignoreHttp1xx": false,
          "port": 8000
        }
    ],
    "cpus": 1,
    "mem": 2048.0,
    "ports": [
        8000
    ]
}

```

<p>Again, here it is crucial to exchange the cassandra and kafka host names for it. Now the cluster contains the applications for accepting the incoming data and for visualizing the data.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/RFhA78ffTDayskniDqi0"><img loading="lazy" width="772" height="206" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/RFhA78ffTDayskniDqi0" alt="dcos_chronos_installation_json" /></a> </p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GFNrnLnATD2P8RwcftgB"><img loading="lazy" width="1277" height="566" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GFNrnLnATD2P8RwcftgB" alt="dcos_marathon" /></a> </p><p>As our data is streaming into the system and we’re ready to actually show that data, it’s crucial to have the spark jobs running. Those spark jobs needed for this platform are installed via the dcos cli.</p><div>

```bash
dcos spark run --submit-args='--supervise -Dspark.mesos.coarse=true --driver-cores 1 --driver-memory 1024M --class de.nierbeck.floating.data.stream.spark.KafkaToCassandraSparkApp https://s3.eu-central-1.amazonaws.com/codecentric/big-data/bus-demo/bus-demo-digest-assembly-0.1.0.jar METRO-Vehicles $CASSANDRA_HOST $CASSANDRA_PORT $KAFKA_HOST $KAFKA_PORT'
```

</div><p>Again make sure you exchange the placeholders with the actual ips and ports.</p><h2>How do we know it is working?</h2><p>Now that everything seems to be working, how do we know it actually does? One possibility is to check if the spark jobs are actually running. For this navigate to the DC/OS overview and select the Apache Spark master. Here a list of running spark jobs is given, the one we just previously deployed should be shown there. After we validated that this job is running, we might want to know how much data is streamed into the system. So you need to find the ip address of the KafkaToCassandraSparkApp driver. Take that ip address with port 4040 and you’re able to see how the data is streamed through the system.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/oKXjV0AtQLy6cD7ly62b"><img loading="lazy" width="1072" height="831" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/oKXjV0AtQLy6cD7ly62b" alt="dcos_spark_runtime" /></a> </p><p>Now that we know the data is actually streamed into the system let’s take a look at the front end.<br /> When navigating to the frontend at :8080 you’ll see the Openlayers frontend where the latest Bus data should be flowing in.</p><div><div><div> <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/rQeTBWhSlasuHcb4YUCc" alt="YouTube" /><div><p>By loading the video, you agree to YouTube’s privacy policy.<br /><a href="https://policies.google.com/privacy?hl=en&gl=en" target="_blank">Learn more</a> </p><p><a href="#">Load video</a> </p><p> <small>Always unblock YouTube</small></p></div></div></div><div></div></div><h2>SMACK Stack DC/OS style …</h2><p>… this Article has shown, doing SMACK the DC/OS style plain rocks. The easiness and readiness of the DC/OS platform give great value to the user.<br /> Just the setup of this showcase takes you about half an hour and you’re ready to go and have your <a href="https://github.com/ANierbeck/BusFloatingData" target="_blank">IoT analytics platform</a>  running. From the application developer view and operations view this platform has shown its readiness.<br /> One part is missing in this article and might find its way into another article in future, it is the automatic setup of such a platform. Within the sources of the showcase you’ll find hints on how one can achieve that with terraform, but it’s out of scope of this article.<br /> DC/OS with an automated setup will give you this application in NO-Time compared to creating a dedicated system on exclusive hardware. This leads us to the next extra you get from running with DC/OS on either on-premise or cloud hosted software. Your Application will scale as easy as this setup by just adding additional nodes.</p><h2>Links:</h2><ul><li>/wissens-hub/blog/iot-analytics-platform</li><li>https://github.com/zutherb/terraform-dcos</li><li>https://github.com/ANierbeck/BusFloatingData</li></ul>

SMACK Stack DC/OS Style

<p>This is going to be a sum-up of the experience gathered on various projects done with the SMACK stack. For details about the SMACK stack you might want to take a look at the following blog – <a href="/wissens-hub/blog/smack-stack-hands" target="_blank">The SMACK Stack – Hands on</a> .<br /> Apache Spark – the S in SMACK – is used for analysis of data – real time data streaming into the system or already stored data in batches.<br /> Apache Mesos – the M in SMACK – is the foundation of the stack. All of the applications do run on it. In our cases we’ve been using Mesospheres DC/OS on top of Apache Mesos for the installation and administration of the stack and our own applications.<br /> Lightbend’s Akka – the A in SMACK – is used for fast data stream processing. In most use cases it’s been either used for fast ingestion of data or for fast extraction through in-stream-processing.<br /> Apache Cassandra – the C in SMACK – is a fast write and read storage for the fast-data-processing platform.<br /> Apache Kafka – the K in SMACK – is the intermediate storage for streaming data. It helps to decouple in and out application logic while still being fast enough to add no overhead of time on the stream of data.<br /> <span><br /> In those projects the architecture has looked roughly like this<br /> <a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/uMsp5jNcRCGH6JDL1AjJ" target="_blank"><img width="1024" height="768" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/uMsp5jNcRCGH6JDL1AjJ" alt /></a> <br /> The ingestion, implemented with Akka, is kind of like Enterprise Integration on steroids. Instead of having a lot of different connectors you’ll end up with just a few entry points, but doing so in a very, very fast way. For each of the projects it had been a requirement to have fast input and storage of the data as well as having that data visible in near real-time. That’s where Akka comes into play again – either being connected to Kafka for real-time streaming of data via Websockets or as connector for Cassandra.<br /> All of the scenarios running this stack have been build on AWS as cloud provider. This made it especially easy to setup and tear down the stack for development.</span></p><h2>DC/OS – Apache Mesos</h2><p>Apache Mesos in combination with Mesospheres DC/OS is the foundation for working with the SMACK stack. <a href="http://mesos.apache.org/" target="_blank">The Mesos<br /> kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments. </a> <br /> On top of Mesos, Mesosphere’s DC/OS will give you a command line interface and a nice integration with Mesosphere’s Marathon. Especially the first one in combination with Ansible can be used to automate the setup of a whole cluster.<br /> The following shell command installs the Cassandra framework used by DC/OS.</p><div>

```bash
dcos package install cassandra
```

</div><p>In combination with the dcos command <strong><em>dcos package list</em></strong> it’s possible to verify the success of the installation.</p><p>While Apache Mesos is used as “Kernel”, Marathon is used as “init.d” on top of it. Marathon makes sure deployed applications are running successfully. In case of a failure those applications are restarted by Marathon. While Marathon takes care of long running tasks, Mesosphere’s Metronome is in charge of short running cron-like tasks.<br /> Marathon and Metronome are so called Mesos frameworks. A framework takes care of reserving resources from Mesos, schedules the application to be launched and sometimes monitors those applications.</p><h2>Apache Cassandra</h2><p>The nice thing about DC/OS is that it is very easy to run Apache Cassandra via a specialized Mesos framework. This framework not only helps in installing a Cassandra cluster on top of Mesos, it also makes sure of handling recovery of failed instances. The configuration and sizing of the Cassandra is crucial for having a high performance fast data platform based on this SMACK stack.</p><h3>Storage</h3><p>For performance reasons it’s best to start with a SSD hard drive, but also <a href="https://d0.awsstatic.com/whitepapers/Cassandra_on_AWS.pdf" target="_blank">EBS volumes</a>  can be used. Using EBS we never experienced any shortcomings, though it seemed we had an increase in write queues. This usually happens if commit log and SSTables are written to the same storage. In cases like these it’s crucial to have a fast connected EBS at hand.</p><h3>CPU</h3><p>In short, more CPU will give you more throughput. As Cassandra uses different thread pools for write and read paths, an increase in number of CPUs helps tremendously, as those thread-pools have more dedicated CPUs.</p><h3>Memory</h3><p>As Cassandra had commodity hardware in mind when being designed, a lot of heap is of no use for a cassandra instance. A lot of RAM is rather useful for the underlying system itself, as the page cache will consume all the memory not used by other applications. Therefore configuring Cassandra with 8GB of RAM as heap plus 24GB for the underlying system for page caches is sufficient. It’s rather crucial to make sure the new generation heap is configured properly. As recommended in the Cassandra documentation the new generation of the heap should be set to 1400MB. Which is equivalent to 100MB times the number of CPUs. The rule of thumb is to either use 100MB times number of CPUs or a quarter of the maximum Heap, where the lesser number needs to be used.<br /> As the system is run with a Java 8 runtime, garbage collection can be set to the GC1 garbage collector.</p><h3>Apache Kafka</h3><p>Like Apache Cassandra and other Big-Data systems, Kafka has also been designed with commodity hardware in mind. Therefore around <a href="http://docs.confluent.io/1.0/kafka/deployment.html#memory">5GB of Heap for the Kafka</a>  process is enough. Again here it’s more important to have enough RAM for the hard drive caches as for Kafka heap usage. Regarding storage for Kafka the same principle applies. SSD should be favored but a fast connected EBS storage is sufficient. Kafka and its thread pools also profit greatly by a higher number of CPUs.<br /> As Cassandra and Kafka in a production use-case are consuming rather lot of cpu and harddrive, one should consider to run those frameworks on dedicated machines. This does break the rule of “every framework is treated equal”, but especially the page caching mechanism only works if those applications are the only user of all resources.</p><h2>Apache Spark</h2><p>Apache Spark is already optimized to run on Apache Mesos. So after the easiness of installing it via a dcos package install spark, spark is ready to be used. Spark comes with a default Mesos scheduler, the MesosClusterDispatcher also known as Spark Master. All spark jobs will register themselves with the master and in turn will also be a Mesos framework. This driver is negotiating with Mesos about the required resources. From there on it takes care about the executors for Spark. Within this scenario of using DC/OS the executors are docker images with a DC/OS optimized configuration. They already contain configurations for access of a HDFS inside the DC/OS cluster.</p><h3>Metrics</h3><p>Spark metrics are nice while being watched in real time, but it would be nice to have the metrics available all the time, especially since after the death of the driver this data is gone. One way is to use the Spark history server. The history server is nice, but requires an HDFS to be available. This alone isn’t much of a downside but the requirements on running HDFS on DC/OS is rather high. At least 5 instances are required with lots of hard-drive. This just for taking a look at the Spark metrics is rather expensive. Therefore a good possibility is to use ELK (ElasticSearch, Logstash and Kibana) for monitoring. But how do we get to the logs of Spark. In a “regular” environment, you’ll usually just add some logging details to the executors, but as our executors are started by mesos and managed by the Spark Driver this needs some extra tweaking in the DC/OS world.</p><p>To enable the spark metrics a configuration is needed as the following:</p><div>

```properties
# Enable Slf4jSink for all instances by class name
*.sink.slf4j.class=org.apache.spark.metrics.sink.Slf4jSink
 
# Polling period for Slf4JSink
*.sink.slf4j.period=1
 
*.sink.slf4j.unit=seconds
```

</div><p>With these settings Spark logs all metrics available to the std logging mechanism. The tricky part is to actually have those settings enabled inside the docker image provided by DC/OS. Actually this isn’t possible, so we built a custom Docker image already containing these settings.<br /> The Dockerfile for such a preconfigured Docker image can be seen below, it’s not much magic.</p><div>

```bash
FROM  mesosphere/spark:1.0.2-2.0.0
 
ADD ./conf/metrics.properties /opt/spark/dist/conf/
```

</div><h3>External Storage</h3><p>As you’ve seen in the previous section, enabling a HDFS system can be quite costly in terms of storage. In this scenario an extra HDFS system isn’t required, therefore storing data in S3 is a quick win. As with the metrics, DC/OS’ own spark image is optimized for HDFS and therefore needs some tweaking for accessing S3.<br /> First, it is crucial to have those S3 accessing libraries available in your docker image. Second, you’ll need to make sure those newly available libraries are present in the configuration of your executor.</p><h4>spark-defaults.conf content:</h4><div>

```properties
spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.executor.extraClassPath /opt/spark/dist/extralibs/aws-java-sdk-1.7.4.jar:/opt/spark/dist/extralibs/hadoop-aws-2.7.2.jar:/opt/spark/dist/extralibs/joda-time-2.9.jar
spark.driver.extraClassPath /opt/spark/dist/extralibs/aws-java-sdk-1.7.4.jar:/opt/spark/dist/extralibs/hadoop-aws-2.7.2.jar:/opt/spark/dist/extralibs/joda-time-2.9.jar
```

</div><p>In the configuration above, there is an additional section setting the filesystem type to S3A which is needed for a faster access to it. The Apache Hadoop driver supports two different ways of accessing S3, one the default S3 is a block based access on the S3, while the S3N or S3A use object based access on S3. Hadoop now only supports S3A as it’s the successor of S3N (native). The extraClassPath entries are needed for the driver and also for the executor. An additional step of creating your own Docker image is to make sure those libraries are available.</p><h3>Using a custom Docker spark image</h3><p>When submitting Spark jobs via DC/OS usually you’ll issue a command as the following:</p><div>

```bash
dcos spark run --submit-args='--driver-cores 0.1 --driver-memory 1024M --class org.apache.spark.examples.SparkPi https://downloads.mesosphere.com/spark/assets/spark-examples_2.10-1.4.0-SNAPSHOT.jar 10000000'
```

</div><p>Now if you want to use your own custom docker image you need to adapt the command to look like the following.</p><p><strong><em>dcos spark run –submit-args =’…’ –docker-image=my-own-docker/spark-driver</em></strong></p><p>In that case you need to make sure this docker image is accessible to your Mesos agents.<br /> Because installing those Spark job can be cumbersome, we used Metronome for an easy installation of those Spark jobs.</p><h2>Throughput</h2><p>After all those technical implications let’s take a look at what can be achieved with such a SMACK platform. Our requirements contained the ability to process approximately 130 thousand messages per second. Those messages needed to be stored in a Cassandra and also be accessible via a frontend for real time visualization.<br /> This scenario is build on the basic architecture. Akka Streams are used for ingestion and also for the real-time visualization. The ingestion does some minor protocol transformation and publishes the data to a Kafka sink. While 4 nodes are capable of handling these 130k msg/s, 8 ingestion instances are needed to keep up with 520k msg/s.<br /> A high throughput in the ingestion is nice, but how much delay does it produce? For example: are those messages queuing somewhere internally Is the back-pressure to high? While being able to handle the throughput of 520k msg/s the delay has an average of 250ms of latency between message creation and storage and Kafka. If the network and ntpd fuzziness is taken into account, the delay produced by the ingest itself is comparable to be nonexistent.<br /> So the ingestion and therefore the visualisation consumer are capable of handling the data in real-time. How does storing the data in Cassandra compare to this?<br /> The easiest use-case for Apache Spark in this scenario is to stream the incoming data into the Apache Cassandra database. While Kafka topics are perfectly suited to be consumed in a streaming way, Cassandra isn’t capable of doing streamed inserts. That’s one of the reasons the largest amount of time is consumed by connecting and communicating with Cassandra. Therefore it’s better to have a batch interval of about 10 seconds. The base-line for connection and minimum time to just get going is about 1 to 2 seconds depending on the underlying hardware and amount of CPUs allocated for the executors. The 10 second window frame also helps handling possible but not envied downtimes of the Spark Job. With this time-buffer it’s possible to handle a longer downtime. Another reason to keep the 10 second window, data might happen to be hold back in the pipeline before consuming in Spark. This will also address this issue of a peak-load.<br /> So let’s talk numbers: a Spark job storing data into Cassandra batched for 10 seconds takes about 4 seconds for 1.3 Million events within the batch window. To store the 5.2 Million events inside the batch window, Spark takes 8 seconds to store those events in Cassandra. As Cassandra is a rather large stakeholder when streaming data from Kafka to Cassandra via Spark, it also needs to scale horizontally with the amount of data. The following table gives a brief overview of amount of messages, average latency per batch and amount of Cassandra nodes needed to handle the amount of messages per second.</p><table><tbody><tr><th>msg/s</th><th>Time needed<br /> Batched 10s</th><th>Cassandra Nodes</th></tr><tr><td>130k msg/s</td><td>4s</td><td>5</td></tr><tr><td>260k msg/s</td><td>6s</td><td>10</td></tr><tr><td>520k msg/s</td><td>8s</td><td>15</td></tr></tbody></table><p>When measuring the throughput of simple events of approximately 60 byte size, the initially upper limit was reached by around 500k msg/s. It turned out to be a network capacity limit of the selected AWS box. This limit of 20MB/s correlated nicely with the approximately 500k msg/s and their corresponding byte size. When selecting another type of box with better network settings as entrypoint for the incoming data, processing 520k msg/s was easily achieved.</p><h3>Conclusion</h3><p>Sometimes working with bleeding-edge software is more like grabbing into falling daggers, on the other hand it’s great to see it turn into an extremely powerful and capable stack. With time progressing, the Mesosphere DC/OS provided functionality turned more stable and more sophisticated. The out-of-the-Box frameworks for the SMACK stack play nicely, and due to those frameworks the stack is resilient concerning failures. A “misbehaving” app will be stopped by the framework while being restarted in the same minute. This failsafe and failure-tolerant behavior gives great confidence in the running cluster as it rarely needs operation to be in charge.<br /> It turned out to be obvious, but this stack scales linearly regarding performance, throughput and used hardware. But not only does it scale up, it also helps to start with a smaller stack. Especially using Apache Mesos as foundation helps to get the most out of your provided resources. This is especially useful when still developing the stack. In this case Apache Cassandra and Apache Kafka may be on the same node, as throughput isn’t the main goal, yet.<br /> While Monitoring is needed it’s still not provided out of the box, a custom solution is needed. For monitoring purposes it’s still required to have your custom solution, with DC/OS 1.10 you’ll most likely will have a solution for <a href="https://github.com/dcos/dcos-metrics" target="_blank">metrics</a> . DC/OS provided Frameworks place a collecting module next to the application, collecting the metrics of that process. Those metrics are locally collected and sent to a central server based on Apache Kafka. This will be an exciting new functionality for measuring metrics of your cluster in future.</p>

SMACK stack from the trenches

<p>Digitization has been revolutionizing each and every business out there for the past few decades. It has surely a lot to offer in your business domain as well: a new customer portal to improve users’ satisfaction and help you reach out to a whole new group of potential customers, an automation platform to replace all the manual, time-consuming, and error-prone procedures in heads of a handful of experts with clearly defined, fully automated, and scalable digital processes, a modern data ingestion and analyzation platform to apply state of the art machine learning and artificial intelligence methods on your wealth of unused data, opening up the doors of new opportunities to your business – You name it!</p><p>As compelling as it sounds, there are often so many organizational complexities associated with new experiments, that make such adventures practically unaffordable for many small or midsize businesses. The computational infrastructure must be carefully planned and provisioned, the cost needs to be calculated and approved, associated risk has to be estimated, and a development team in possession of the required expertise needs to be established – to name just a few. Many cycles of planning, development, testing, and measuring need then to pass on before any customer insights can flow in.</p><p>The delayed nature of customer feedback in such constellations associates a high chance of failure to such approaches, due to a misunderstanding of the actual problem or misinterpretation of the end-users’ requirements.</p><h2><span>Developing user-centric software products fast and reasonable</span></h2><p><span>Solution Factory is a user-centric, fast, and reliable approach towards experimenting with new ideas and developing ground-breaking, innovative solutions. The idea and the user experience play the most central roles, around which everything else evolves and gets evaluated. </span><span>In tight collaboration with domain experts and end-users, the customers are guided through the whole journey of confining the problem at hand, building a Minimum Viable Product (MVP), and evolving the MVP into a fully-fledged end-product. </span></p><p><span>Solution Factory is composed of three stages: Design Sprint, MVP Development, and Solution Development.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/m57d35GATBOu4ji6UNOI"><img width="250" height="175" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/m57d35GATBOu4ji6UNOI" alt /></a> </p><h2><span>Design Sprint</span></h2><p><span>The concept of a Design Sprint is based on the lean startup methodology. This methodology allows us to design disruptive business cases while keeping track of the risks in parallel. We define a business hypothesis (idea) and translate (build) it into code. Then we measure customer’s behavior (measure) and collect findings (data). This gives us an idea, which we can optimize afterwards (learn). This way, customers can be better understood and are integrated in the process of defining new hypotheses. This produces a conclusive chain of proof – of what works and what does not. Through iterative software development, new ideas can be quickly tested and “bad” ideas are discarded without losing customers or budget.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/t5NhXnm4SO24dUMf4bi2"><img loading="lazy" width="250" height="249" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/t5NhXnm4SO24dUMf4bi2" alt /></a> </p><p><span>With new products, however, there is a so-called </span><span>“The chicken or the egg?”</span> <span>causality dilemma</span><span>: Without a ‘theoretical’ solution, no data can be collected, and without data, no new hypotheses can be generated. The Design Sprint approach takes a short circuit to bring end-users, domain experts, and stakeholders together, draw a clear picture of the extent of the problem at hand, prototype a solution, and learn from end-users before a single line of code is written. In the language of the lean software development, it learns directly from the business hypothesis (ideas), without much of data gathering.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/t2kU0yR9QwaJI9PGcvj6"><img loading="lazy" width="250" height="249" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/t2kU0yR9QwaJI9PGcvj6" alt /></a> </p><h2><span>MVP Development </span></h2><p><span>MVP development stage starts right after the Design Sprint. It consists of six agile sprints of software development, one week each. Every sprint starts on Monday and ends with a review meeting on Friday, where new features are presented and the next sprint is planned. This stage ends after the sixth sprint, producing an MVP that is handed over to the customer. </span><span>The MVP is a working application that implements exemplified key aspects of the final solution and is useable by end-users. There are then two weeks of break</span><span>, during which the MVP is thoroughly evaluated by the customer. Also, the decision is made, whether to proceed to the next stage or to rework the hypotheses and iterate over the idea or the MVP.</span></p><h2><span>Solution Development</span></h2><p><span>Solution Development stage is to convert an approved MVP into a final solution and bring it into production. The objectives at this stage are to find out what inherently distinguishes solution from MVP development. With MVP, the goal is to build a minimal set of features that enable the customer to experiment with the idea itself. To achieve this goal during the limited period of six weeks, a healthy amount of technical debt is consciously allowed in the MVP. In Solution Development after that, not only new features are developed, but also the technical debts are iteratively reduced and the MVP is grown up into a production-ready solution.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2ZnoNcHsTqyMgBazuh63"><img loading="lazy" width="250" height="175" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2ZnoNcHsTqyMgBazuh63" alt /></a> </p><h1><span>Success Story</span></h1><p><span>Judging from our experience, Solution Factory is one of the most effective models for approaching new projects and experimenting with new ideas at minimum risk and maximum customer satisfaction. In the following, read about how Solution Factory helped one of our customers to tackle major problems in its customer onboarding process. </span></p><h2><span>Problem</span></h2><p><span>Our customer, one of the biggest in its sector in Germany, is legally obliged to go through a complicated onboarding process for new customers: various pieces of information need to be gathered, different legal aspects need to be secured, and contracts need to be signed between multiple service providers. The process spans over multiple participating beneficiaries and takes many weeks before it can eventually be finalized.</span></p><p><span>To overcome all these complications, the customers had to be taken care of in person. They used to be walked through the complete process by an employee, who continuously had to track the progress and keep contact with the customer, reminding him of missing information and the outstanding deadlines. On the one hand, this manual process used to extremely increase the effort on our customer’s side. On the other hand, the inefficient communication and the confusion associated with it used to slow down the entire process,  </span><span>increase error rate, and damage customer satisfaction.</span></p><h2><span>Solution</span></h2><p><span>A group consisting of a Design Sprint coach, a user experience (UX) expert, a user interface (UI) designer, and a software and cloud architect was established. The customer’s technical experts, as well as a representative group of end-users, were invited to join us for a week of Design Sprint in our codecentric office in Karlsruhe.</span></p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/3HDCLsaxRza9rac0Ah6j"><img loading="lazy" width="250" height="175" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/3HDCLsaxRza9rac0Ah6j" alt /></a> </p><p><span>Monday was dedicated to understanding the domain and the problem at hand. The domain experts were asked to describe how they evaluated the current situation and where they saw the biggest pain points. The team actively participated in the discussion by asking questions and gathering points. The outcome was then processed in the group and formalized into a concise description, which defined the target of the Design Sprint. The collaborative work on monday proved to be extremely valuable for the whole team, as it clearly defined the scope of the Design Sprint and helped to establish a common language between the domain and the IT experts.</span></p><p><span>Having identified the problem , Tuesday was devoted to exploring the solution space. Each participant was asked to gather ideas and draw sketches of how a potential solution might look like. These sketches defined the building blocks of the final solution, which was yet to be found. They were presented to the team at the end of the day, leaving out the discussion and evaluation for the next day. On Wednesday and Thursday, the sketches were evaluated and the ideas behind them were combined into a final proposal. The proposal was then converted into a click-dummy prototype which mimicked the complete user journey without any real implementation. </span></p><p><span>Friday was the last and the most exciting day of our intensive week of domain knowledge transfer, discussion, and ideation. Although we had a proposal at hand, there still was no guarantee that it would really meet the requirements of the end-users and fix the initial problem. On Friday, five separate interviews were organized with target-users of the product. They were well-aware of the current situation and its deficiencies and were about to see our proposed solution for the first time. Each interview took around 45 to 60 minutes. An interviewer described one or more tasks that we wished to have facilitated. The interviewee was then asked to try and achieve the tasks using the prototype while thinking loudly, describing what she perceived, and stating if the perception matched her expectation. In the end, the user was asked to rate her overall experience. The whole interview was carefully observed by the team. </span></p><p><span>On a scale from 1 to 10, with 1 being the least and 10 the most satisfying, our prototype achieved an average of 8.5. Interestingly, all interviewees asked about when the product would be available at the end of their interviews. This was clearly a success and the green light for the transition to the next stage.</span></p><h2><span>Result</span></h2><p><span>A customer management portal is conceived. It models the complete onboarding process of new customers and enables users to self-manage their products at any time without the need for manual intervention. The user can request new contracts, track the progress of the existing ones, and determine their next open tasks to fulfill. The portal tracks the progress of all contracts and walks customers through all necessary steps. Automatic reminders ensure that no deadlines will ever be overseen by any participating party.</span></p><p><span>During the six upcoming weeks after the Design Sprint, an MVP was developed at codecentric AG in tight collaboration with the customer’s domain experts. The MVP was then evaluated and has been further developed into a final solution over the course of the past twelve months. The development team consisted of a software developer and architect from Karlsruhe office, together with two full-stack developers from codecentric’s Banja Luka office in Bosnia, and a designer from codecentric partner UX&I from Düsseldorf. For an optimum team building, the bosnian colleagues spent the first three weeks of MVP development in Karlsruhe, and have been working remotely since then. They were in constant sync with the rest of the team through ad-hoc communication and regular meetings. Despite the distributed nature of communication, the teamwork proved to be effective and the performance was consistently high. Thanks to cloud-native and serverless architecture with continuous delivery, the stakeholders were tightly involved in the complete development process, feedback cycles were extremely reduced. Also, infrastructure and operation costs were minimized.</span></p><h1><span>Conclusion</span></h1><p><span>Classical software development is often very slow in terms of getting customer feedback and adjusting the outcome accordingly. This increases the risk and the cost of conventional approaches enormously when generating hypotheses and experimenting with new ideas. Solution Factory offers a new approach to developing fast and reliable user-centric software products. It boosts disruptive innovation inside organizations by minimizing the risk, cost, and feedback lifecycle. </span></p><p><span>If you want to learn more about our concept Solution Factory and how it might help your organization, feel free to contact us.</span></p><p> </p>

Solution Factory – How to get from idea to product in 9 weeks

IoT Analytics Platform

Niederlassungsleiter

Office Manager Region Südwest

Key Account Manager

<h2>Was ist Terraform und was ist State?</h2><p>Terraform ist ein Tool für die Verwaltung von Infrastruktur in Form von Code, gehört also in den sogenannten <em>Infrastructure-as-Code-</em>Bereich (IaC). Eine kurze Einführung und ein Vergleich zu anderen Tools findet sich <a href="/wissens-hub/blog/infrastructure-as-code">hier</a> .</p><p>Terraform verwendet intern einen State (persistiert als JSON-Datei), der den aktuellen Zustand der Infrastruktur enthält. Dieser State beschreibt den Zusammenhang zwischen den Ressourcen beim Provider und den im Skript verwendeten Ressourcen. Er wird bei jedem erfolgreichen Deployment aktualisiert. Mittels <em>terraform import</em> kann man auch bereits vorhandene, also nicht mittels Terraform erzeugte, Infrastruktur zum State hinzufügen. Möchte man die Infrastruktur ändern, so macht man Änderungen im Terraform-Code, was den neuen gewünschten Zustand darstellt. Terraform vergleicht diesen neuen Zustand dann mit dem bisherigen und kann dadurch die benötigten Änderungen berechnen, was zu effizienten Deployments führt.</p><p><img width="1024" height="576" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4n4uY5j3ThKAbFDApflM" alt="Diagramm das erklärt wie Terraform Infrastruktur managed" /></p><p>Da State so ein zentraler Bestandteil von Terraform ist, gilt es, diesen entsprechend zu verwalten. Dabei geht es vor allem um Schutz vor unbefugtem Zugriff und Datenverlust.</p><h2>Was ist Remote State und warum brauche ich das?</h2><p>Standardmäßig wird der State von Terraform als Datei lokal – also dort, wo Terraform ausgeführt wird – vorgehalten. Dies ist zwar für einzelne Entwickler oder Tests geeignet. Für die Verwaltung eines großen Produktivsystems kann es jedoch sehr problematisch sein, da der State sehr anfällig für Datenverlust ist und man Änderungen genau nur dort durchführen kann, wo sich die State-Datei findet. Ebenso können sich sensible Informationen wie Passwörter im State befinden, weshalb es z. B. keine gute Idee wäre, die State-Datei in ein VCS einzuchecken.</p><blockquote><p>Generell sollte die State-Datei als sensible Information betrachtet werden und es sollten entsprechende Schutzmaßnahmen durchgeführt werden!</p></blockquote><p>Um diesen Problemen zu begegnen, bietet Terraform den sogenannten <em>Remote State</em> an. Hierbei wird der State an einer zentralen Stelle vorgehalten, die im Skript definiert wird, so dass man Deployments von überall ausführen kann, von wo aus auf diese Stelle zugegriffen werden kann. Beliebte Backends hierfür sind AWS S3 oder Azure Blob Storage, die Terraform neben weiteren unterstützt.</p><p>Um Remote State zu verwenden, muss das Backend lediglich im Terraform-Skript definiert werden, hier am Beispiel von S3:</p><div>

```yaml
terraform {
  backend “s3” {
    bucket = “myremotestatebucket”
    key = “path/to/my/state.tfstate”
    region = “eu-central-1”
    encrypt = true
  }
}
```

</div><p>Bei der Verwendung von Remote State können natürlich Race Conditions auftreten, z. B. wenn mehrere Deployments parallel laufen und dabei den State verändern möchten. Um dies zu verhindern, kann man in AWS eine DynamoDB-Tabelle als Lock verwenden. Die Backend-Konfiguration wird dafür um den Verweis auf die Tabelle erweitert:</p><div>

```yaml
terraform {
  backend “s3” {
    bucket = “myremotestatebucket”
    key = “path/to/my/state.tfstate”
    region = “eu-central-1”
    dynamodb_table = “mylocktable”
  }
}
```

</div><h2>Wie kann Remote State noch verwendet werden?</h2><p>Terraform bietet die Möglichkeit, Remote State als Datenquelle zu verwenden. Dies ermöglicht es, in Konfigurationen auf die Ergebnisse anderer Konfigurationen zuzugreifen. Dieser Mechanismus versetzt den Nutzer etwa in die Lage, die Konfiguration eines größeren Gesamtsystems zu dezentralisieren. Eine Variante ist es etwa, gemeinsam verwendete, zentrale Komponenten wie Domains, Zertifikate oder Datenbanken in eine „Infrastruktur“-Konfiguration auszulagern. Die anderen Teile des Systems wie etwa einzelne Microservices können dann in ihren eigenständigen Konfigurationen über den Remote State auf die benötigten Komponenten zugreifen.</p><h3>Beispiel</h3><p>Eine SaaS-Applikation, bestehend aus mehreren Komponenten, darunter ein Microservice, der als Lambda implementiert ist, sollen in einem VPC deployt werden. Um dem Microservice-Team die Verantwortung für sicherheitskritische Dinge wie VPC, Subnets oder Security Groups abzunehmen, übernimmt diese das zentrale Deployment:</p><div>

```yaml
terraform {
  backend “s3” {
    bucket = “centraldeploymentstate”
    key = “global/terraform.tfstate”
    region = “eu-central-1”
    dynamodb_table = “locktable”
  }
}
 
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
 
  tags = local.common_tags
}
 
resource "aws_subnet" "lambda" {
  vpc_id 	= aws_vpc.main.id
  cidr_block = "10.0.1.0/24"
 
  tags = local.common_tags
}
 
resource "aws_security_group" "allow_tls" {
  name    	= "allow_tls"
  description = "Allow TLS inbound traffic"
  vpc_id  	= aws_vpc.main.id
 
  ingress {
    description  	= "TLS from VPC"
    from_port    	= 443
    to_port      	= 443
    protocol     	= "tcp"
    cidr_blocks  	= [aws_vpc.main.cidr_block]
  }
 
  egress {
    from_port    	= 0
    to_port      	= 0
    protocol     	= "-1"
    cidr_blocks  	= ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
 
  tags = local.common_tags
}
 
resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda"
 
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
	{
  	"Action": "sts:AssumeRole",
  	"Principal": {
    	"Service": "lambda.amazonaws.com"
  	},
  	"Effect": "Allow",
  	"Sid": ""
	}
  ]
}
EOF
}
 
resource "aws_iam_role_policy_attachment" "lambda_role_vpc_execution" {
  role = aws_iam_role.iam_for_lambda.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}
```

</div><p>Damit die Microservice-Konfiguration die benötigten Ressourcen verwenden kann, definieren wir diese noch als Outputs:</p><div>

```yaml
output "vpc_id" {
  value   	= aws_vpc.main.id
  description = "ID of the main VPC"
}
 
output "lambda_subnet_id" {
  value   	= aws_subnet.lambda.id
  description = "ID of the subnet dedicated to Lambdas"
}
 
output "tls_security_group_id" {
  value   	= aws_security_group.allow_tls.id
  description = "ID of the allow_tls security group"
}
 
output "lambda_iam_role" {
  value   	= aws_iam_role.iam_for_lambda.arn
  description = "ARN of Lambda execution role"
}
```

</div><p>Das Microservice-Team definiert sich nun den Remote State der zentralen Konfiguration als Datenquelle, um auf Dinge wie die VPC ID zugreifen zu können:</p><div>

```yaml
data "terraform_remote_state" "global" {
  backend = "s3"
  config = {
    bucket = "centraldeploymentstate"
    key	   = "global/terraform.tfstate"
    region = "eu-central-1"
  }
}
```

</div><p>In der Konfiguration des Lambda können wir die zentral verwalteten Ressourcen dann einfach verwenden, ohne manuell IDs oder Ähnliches kopieren zu müssen:</p><div>

```yaml
resource "aws_lambda_function" "test_lambda" {
  filename  	   = "lambda.zip"
  function_name    = "TestLambda"
  role      	   = data.terraform_remote_state.global.outputs.lambda_iam_role
  handler   	   = "index.js"
  source_code_hash = filebase64sha256("lambda.zip")
 
  runtime = "nodejs12.x"
  vpc_config {
    subnet_ids     	= [data.terraform_remote_state.global.outputs.lambda_subnet_id]
    security_group_ids  = [data.terraform_remote_state.global.outputs.tls_security_group_id]
  }
}
```

</div><h3>Vorteile</h3><p>Bestimmte zentrale Komponenten und Services in einer separaten Konfiguration zusammenzufassen, verringert die Wahrscheinlichkeit von unabsichtlichen Veränderungen und dadurch entstehende Ausfälle.<br /> Zudem müssen sich die Teams der einzelnen Komponenten wie Microservices nicht mit den Besonderheiten für die Konfiguration der zentralen Dienste auskennen, sondern können sich auf die Konfiguration ihres Services konzentrieren.<br /> Zusätzlich können wir auf diese Art und Weise eine gewisse Standardisierung und Wiederverwendung erreichen, z. B. von Policies und Rollen für bestimmte Aufgaben oder gemeinsames zentralisiertes Logging.<br /> Schließlich erhöht die Verwendung getrennter States die Performance von Terraform, da bei jedem Deployment weniger Ist-Zustand mit neuem Soll-Zustand abgeglichen werden muss – im Gegensatz z. B. zu dem Ansatz, jedes Mal das gesamte System neu zu deployen.</p><h3>Nachteile</h3><p>Wie immer gibt es natürlich auch hier Nachteile bzw. Dinge, die beachtet werden müssen. Zentral ist natürlich die Frage, wie man die einzelnen Konfigurationen schneidet, also welche Komponenten man gemeinsam deployt und im State hält. Als Richtlinie bieten sich hier vor allem die Abhängigkeiten an. Zyklen müssen durchbrochen werden, sonst klappt das Deployment erst gar nicht.<br /> Wichtig zu beachten ist auch die Kompatibilität zwischen State-Datei und verwendeter Terraform CLI-Version. Wird z. B. das zentrale Deployment mit einer neuen Version aktualisiert, kann es sein, dass ein Komponenten-Deployment mit älterer Version den Remote State nicht mehr lesen und dann fehlschlagen kann. Seit Version 1.0 ist das State-Format jedoch eingefroren, so dass hier immer Rückwärtskompatibilität gewährleistet ist.<br /> Schließlich benötigt für diesen Ansatz jede Konfiguration vollen Zugriff auf den Remote State, der als Abhängigkeit verwendet wird. Dies kann einen Minimum-Required-Access-Ansatz erschweren.</p><h3>Alternativen</h3><p>Auch wenn die Verwendung von Remote State zum Teilen von Konfigurationsinformation einfach und schnell ist, gibt es doch Gründe, Alternativen zu verwenden. Die bereits beschriebene Zugriffskontrolle auf den State ist ein gewichtiger Grund, aber z. B. auch die Anforderung, dass andere Systeme (nicht nur Terraform) auf Konfigurationsinformationen zugreifen können sollen. So existieren z. B. in AWS für quasi alle Ressourcen auch entsprechende Datenquellen, die es ermöglichen, nach bestimmten existierenden Ressourcen zu suchen und Ergebnisse zu filtern. So kann etwa nach Subnets gesucht und aus dem Ergebnis können die CIDR-Blöcke exportiert werden:</p><div>

```yaml
data "aws_subnets" "example" { 
  filter { 
    name   = "vpc-id" 
    values = [var.vpc_id] 
  } 
} 
 
data "aws_subnet" "example" { 
  for_each = toset(data.aws_subnets.example.ids) 
  id       = each.value 
} 
 
output "subnet_cidr_blocks" { 
  value = [for s in data.aws_subnet.example : s.cidr_block] 
}
```

</div><h3>Wann also sollte ich Remote State zur Dezentralisierung verwenden?</h3><p>Auf diese Frage gibt es leider keine pauschale Antwort. Man kann jedoch sagen, dass sich das oben beschriebene Beispiel anbietet, wenn ein Entwicklungsteam das Deployment des gesamten Systems und damit auch den gesamten Terraform State kontrolliert. In diesem Fall kann sich ein kleiner Teil des Teams um die grundlegende Infrastruktur kümmern und z. B. die Frontend- oder Microservice-Entwickler jeweils nur um ihre eigenen Konfigurationen. Gerade bei der Entwicklung von POCs oder eines MVP kann dadurch die Entwicklungsgeschwindigkeit erhöht und gleichzeitig eine solide Basis für das Infrastruktur-Management geschaffen werden.</p><p>Natürlich gilt es immer zu prüfen, ob die spezifischen Datenquellen für die Ressourcen oder wiederverwendete Terraform-Module nicht auch zum gewünschten Ziel führen können. Denn es kann nicht genug betont werden, dass der State sensible Informationen enthält!</p><p>Schließlich noch ein Beispiel aus der Praxis von meinem Kollegen Dennis Grunert. Er verwendet Terraform State, um einen Load Balancer vor „Script Kiddies“ zu schützen. Dazu konfiguriert er die Firewall so, dass sie nur Requests durchlässt, die einen speziellen Header mit einem geheimen Wert enthalten. Diesen geheimen Wert erzeugt man beim Deployment in Terraform mittels der Funktion <code>random_password()</code> und gibt sie sowohl der Firewall als auch dem CDN mit. Neben Terraform und seinem State ist hier nichts weiter erforderlich, um diesen geheimen Wert zu verteilen und vorzuhalten.</p><h2>Fazit</h2><p>Remote State ist ein essenzielles Konzept für die produktive Verwendung von Terraform zur Verwaltung von Infrastruktur-Ressourcen. Neben den offensichtlichen Vorteilen der Datensicherheit und -verfügbarkeit bietet es durch die Möglichkeit, auch als Datenquelle zu fungieren, weitreichende Optionen, komplexe Gesamtsysteme modular abzubilden.</p><h2>Referenzen</h2><ul><li><a href="https://www.terraform.io/language/settings/backends" target="_blank">Remote state offizielle Dokumentation</a> </li><li><a href="https://www.terraform.io/language/state/remote-state-data" target="_blank">Remote state data source offizielle Dokumentation</a> </li><li><a href="https://github.com/nozaq/terraform-aws-remote-state-s3-backend" target="_blank">S3 remote state Terraform Modul</a> </li></ul>

Terraform Remote State richtig nutzen

Senior Marketing Manager

Lead graphic and multimedia designer

Bei der Softwareentwicklung ist es sinnvoll, bereits bestehende Funktionen wiederzuverwenden. Das spart Zeit und es wird unwahrscheinlicher, auf Probleme zu stoßen, die andere bereits gelöst haben. Funktionen können aus diesem Grund in Libraries gebündelt werden. Solche Libraries findet man in Repositories, aus denen man sie mit Paketmanagern beziehen und in die eigene Anwendung installieren kann. Durch die Nutzung von Libraries kommt man schneller und damit kostengünstiger zu einem Ergebnis und muss dabei nicht auf Qualität verzichten. Hierfür ist es jedoch auch notwendig, die verwendeten Libraries sorgfältig auszuwählen, um die eigenen Qualitätsansprüche sicherzustellen.  

Libraries stehen außerdem nicht still, sondern werden kontinuierlich weiterentwickelt. Aus diesem Grund werden sie mit einer Versionsnummer im Repository veröffentlicht und von einem Paketmanager in einer bestimmten Version (meistens der aktuellsten) in der eigenen Anwendung installiert. Wird eine neue Version veröffentlicht, kann dies folgende Gründe haben:

- **Sicherheit**  
  Sicherheitslücken werden immer wieder gefunden und müssen geschlossen werden. Selbst bekannte und viel genutzte Bibliotheken weisen Sicherheitslücken auf, wie die Bibliothek Log4j gezeigt hat.
- **Performance**  
  Oft lassen sich Funktionen in ihrer Performance optimieren, sodass sie schneller laufen oder weniger Ressourcen (CPU, Speicher) benötigen.
- **Features**  
  Neue Features kommen zur Software hinzu, um näher an das verfolgte Ziel zu gelangen. Genauso können Features entfernt werden, weil sie durch neue ersetzt oder nicht mehr benötigt werden.

Eine Anwendung beinhaltet heutzutage schnell einige Dutzend Bibliotheken, auch Dependencies genannt. Diese veröffentlichen regelmäßig neue Versionen, zum Teil mehrmals pro Woche. Das liegt oft auch daran, dass Dependencies selbst wiederum Dependencies verwenden und so ein Abhängigkeitsbaum entsteht, der einer Anwendung in JavaScript schnell über 1000 Bibliotheken hinzufügt.

Um zeitnah von den Vorteilen einer neuen Version zu profitieren, sollten Dependencies regelmäßig geupdatet werden. Auch gilt, umso älter Dependencies sind, desto aufwendiger wird es, diese auf die aktuelle Version zu updaten.

Das liegt daran, dass das Changelog wächst und die Wahrscheinlichkeit für Breaking Changes zunimmt. Um Fehler, die durch Breaking Changes auftreten können, aufzudecken, sollten mit jedem Update Regressionstests durchgeführt werden.

## Automatisierung der Depency-Updates mit Renovate

Renovate ist ein Tool, welches das Updaten der Dependencies übernehmen kann. Dabei lässt es sich sehr einfach in CI-Pipelines integrieren. Renovate unterstützt viele <a href="https://docs.renovatebot.com/#:~:text=Renovate%20Modules-,Language%20Support,-Deep%20Dives" target="_blank">Sprachen</a> und <a href="https://docs.renovatebot.com/#supported-platforms" target="_blank">Plattformen zur Codeverwaltung</a>.

## Funktionsweise

Wenn Renovate eine neue Version einer Dependency findet, wird ein neuer Branch angelegt und die Version dort geupdatet. Danach laufen die konfigurierten Pipeline-Jobs auf diesem Branch. Diese sollten unter anderem Regressionstests enthalten, die testen, ob die Anwendung auch unter der neuen Version weiterhin funktioniert wie erwartet. Neben dem Branch erstellt Renovate einen Merge-Request auf den Haupt-Branch. In dem Merge-Request zeigt Renovate die Änderungen aus dem Changelog der Dependency an. Entscheidet man sich, eine Dependency zu updaten, kann der Merge-Request geschlossen werden und das Update fließt so in die Anwendung ein. Eine Einstellmöglichkeit erlaubt, dass Änderungen nach erfolgreich gelaufener Pipeline automatisch gemerget werden. Dies sollte nur geschehen, wenn die Anwendung mit ausreichend Tests versehen ist. Neben den Merge-Requests gibt es noch einen Issue, der als <a href="https://docs.renovatebot.com/key-concepts/dashboard/" target="_blank">Dashboard</a> den Zustand aller Dependencies anzeigt. Mit dem Dashboard lassen sich auch Freigabeprozesse für bestimmte Updates einrichten. Wie zum Beispiel <a href="https://docs.renovatebot.com/key-concepts/dashboard/#require-approval-for-major-updates" target="_blank">Major Updates</a>, <a href="https://docs.renovatebot.com/key-concepts/dashboard/#require-approval-for-specific-packages" target="_blank">spezielle Dependencies</a> oder für <a href="https://docs.renovatebot.com/key-concepts/dashboard/#require-approval-for-all-updates" target="_blank">alle Updates</a>.

## Einrichtung in GitLab: Aufsetzen des RenovateBot

### Access Token

Es gibt verschiedene Wege, wie Renovate auf ein Repository oder mehrere Repositories zugreifen kann. Wenn man Renovate nur für ein Repository benutzen will, reicht ein GitLab-Accesstoken mit *api*- und *write_repository*-Zugriff. Soll sich Renovate um mehrere Repositories kümmern, empfiehlt es sich, einen Renovate-Benutzer in GitLab anzulegen und diesen als Member (Maintainer oder Developer) den relevanten Projekten hinzuzufügen. Diese Variante wird im folgenden Beispiel verwendet.

### .gitlab-ci.yml

Dazu wird ein neues Repository *renovate* angelegt, welches in der *.gitlab-ci.yml* folgenden Renovate-Job enthält:

```yml
stages:
  - renovate

renovate:
  image: renovate/renovate:slim
  stage: renovate
  variables:
    RENOVATE_PLATFORM: gitlab
    RENOVATE_ENDPOINT: $CI_API_V4_URL
    RENOVATE_AUTODISCOVER: "true"
    RENOVATE_BINARY_SOURCE: install
    LOG_LEVEL: debug
  tags:
    - docker
  only:
    - schedules
  script:
    - renovate $RENOVATE_EXTRA_FLAGS
```

In diesem Beispiel läuft Renovate in einem Docker-Container, welcher von einem <a href="https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#register-a-runner-that-uses-the-docker-executor" target="_blank">GitLab Runner (Docker executor)</a> gestartet wird.

Damit der Job laufen kann, werden folgende Environment-Variablen benötigt:

- <a href="https://docs.renovatebot.com/modules/platform/gitlab/" target="_blank">RENOVATE_TOKEN</a> _(CI/CD Variable)_  
  Hierfür wird für den Renovate-User ein Access Token angelegt, welcher *api*- und *write_repository*- Zugriff für GitLab erhält. Der *api*-Zugriff wird benötigt, damit Renovate den Dependency-Issue updaten, sowie Merge-Requests anlegen kann. Der *write_repository*-Zugriff wird benötigt, damit Dependency Files (z. B. package.json) geupdatet werden können.
- <a href="https://docs.renovatebot.com/getting-started/running/#githubcom-token-for-release-notes" target="_blank">GITHUB_COM_TOKEN</a> _(CI/CD Variable)_  
  Hier wird ein *readonly*-GitHub-Access-Token eines beliebigen GitHub-Accounts benötigt, um Release Notes für die auf GitHub veröffentlichten Dependencies zu laden.
- <a href="https://docs.renovatebot.com/self-hosted-configuration/#platform" target="_blank">RENOVATE_PLATFORM</a> _(.gitlab-ci.yml)_  
  Hier wird angegeben, dass Renovate mit GitLab läuft.
- <a href="https://docs.renovatebot.com/self-hosted-configuration/#endpoint" target="_blank">RENOVATE_ENDPOINT</a> _(.gitlab-ci.yml)_  
  Hier wird die API-URL von GitLab angegeben. Diese kann aus der von GitLab gesetzten Umgebungsvariable *CI_API_V4_URL* bezogen werden.
- <a href="https://docs.renovatebot.com/self-hosted-configuration/#autodiscover" target="_blank">RENOVATE_AUTODISCOVER</a> _(.gitlab-ci.yml)_  
  Dieses Flag gibt an, dass Renovate automatisch alle Repositories, auf die es Zugriff hat, nach Dependencies durchsuchen soll.
- <a href="https://docs.renovatebot.com/self-hosted-configuration/#binarysource" target="_blank">RENOVATE_BINARY_SOURCE</a> _(.gitlab-ci.yml)_  
  Damit installiert sich Renovate “third-party tools”, die es benötigt, um Updates durchführen zu können, z. B npm, yarn, …

Es gibt noch viele weitere <a href="https://docs.renovatebot.com/self-hosted-configuration" target="_blank">Einstellungsmöglichkeiten</a>, die über diese Basiskonfiguration hinausgehen.

### Scheduling

In der *.gitlab-ci.yml* ist unter *only* angegeben, dass Renovate über einen GitLab-Pipeline-Schedule gestartet wird. Dieser wird im nächsten Schritt angelegt:

![gitlab-schedule.png](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Ua56uJzHRtqah8dhTCTg)

Über das *Interval Pattern* wird angegeben, wie oft Renovate laufen soll. Es gibt keine allgemeine Empfehlung, wie oft dies geschehen sollte. Läuft es zu oft und gibt es sehr viele Dependency-Updates, kann es den normalen Entwicklungsprozess stören. Aus diesem Grund empfiehlt es sich, Renovate außerhalb der üblichen Arbeitszeiten laufen zu lassen. Renovate sollte aber auch nicht zu selten laufen, da das Mergen eines Update-Branches mit anschließendem Rebase eines anderen Update-Branches nicht in einem Durchlauf erfolgt.

Damit ist die allgemeine Basiskonfiguration von Renovate für eine GitLab-Instanz abgeschlossen.

## Projektkonfiguration

Um Renovate in einem Repository zu aktivieren, genügt es, den Renovate-Nutzer als Maintainer oder Developer dem Repository hinzuzufügen. Da die Umgebungsvariable *Autodiscover* im Renovate-Job auf *true* gesetzt ist, wird Renovate beim nächsten Durchlauf das Repository automatisch finden und analysieren.
Zudem ist es sinnvoll, im Projekt eine *renovate.json* Datei anzulegen, in der die <a href="https://docs.renovatebot.com/configuration-options/" target="_blank">Konfiguration</a> für das Repository ergänzt werden kann.

```json
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base", ":pinAllExceptPeerDependencies"],
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch"],
      "automerge": true
    }
  ]
}
```

In der *extends*-Property werden sinnvolle <a href="https://docs.renovatebot.com/presets-config/#configbase" target="_blank">Presets</a> angegeben. Mit *:pinAllExceptPeerDependencies* werden alle Dependencies auf eine feste Version in der package.json-Datei festgeschrieben. Oft schreibt z. B. npm die Versionsnummer mit einem <a href="https://github.com/npm/node-semver#caret-ranges-123-025-004" target="_blank">Carret Range</a> in die package.json. Dies ist aber nicht mehr nötig, da Renovate immer für die aktuelle Version sorgt.

Daneben wird noch eine PackageRule angelegt, die dafür sorgt, dass alle Minor- und Patch-Updates automatisch gemerget werden, wenn die Pipeline mit den Tests durchgelaufen ist.

## Fazit

Renovate kann mit wenig Konfiguration schnell eingesetzt werden, um Dependencies regelmäßig zu updaten. Damit automatisiert es regelmäßig auftretende Aufgaben und hilft, die Anwendung immer aktuell zu halten. Dies trägt zum fehlerfreien und zuverlässigen Betrieb der Anwendung bei und steigert damit die Wertschöpfung.

Automatische Dependency-Updates mit Renovate

<p>In this blog post we explain how you can build your own face detection application without much machine learning knowledge. Why? At codecentric everyone has one day per week for professional development and training. Among other things we use this time to get in touch with new technologies and build cool stuff. This time we decided to have a closer look at the Coral USB accelerator. You can see the outcome in the following video.</p><div><div><div> <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/zotbwjpLTLe66w2oc1du" alt="YouTube" /><div><p>By loading the video, you agree to YouTube’s privacy policy.<br /><a href="https://policies.google.com/privacy?hl=en&gl=en" target="_blank">Learn more</a> </p><p><a href="#">Load video</a> </p><p> <small>Always unblock YouTube</small></p></div></div></div><div></div></div><p>The application detects faces based on a pre-trained neural network and overlays them with face filters. In order to keep the face filters assigned to individual faces, even if multiple people appear in the video, it tracks the detected faces over time. In this blog post we explain how it works and how you can build your own face detection application with low cost consumer hardware and without much machine learning knowledge.</p><h2>Our hardware setup</h2><p>We used the following hardware components:</p><ul><li><a href="https://coral.withgoogle.com/products/accelerator/" target="_blank">Coral USB Accelerator</a> </li><li><a href="https://www.raspberrypi.org/products/raspberry-pi-4-model-b/" target="_blank">Raspberry PI 4</a> </li><li>A webcam with 60fps – <a href="https://elinux.org/RPi_USB_Webcams" target="_blank">have a look at webcams which are compatible with a Raspberry Pi</a> </li></ul><p>The Coral USB accelerator connected to a Raspberry Pi 4 is the heart of the setup. The accelerator contains an edge TPU (Tensor Processing Unit) coprocessor which is optimized to process matrix operations. It currently only supports pre-compiled TensorFlow Lite models. It can perform 4 trillion operations per second. Therefore, it allows high inference speed for image classification and object detection using neural networks. When we ran our experiments on the CPU of the Raspberry Pi 4 without the Coral USB accelerator, the application could process between 0.5 and 1.5 frames per second. Using the accelerator, we achieved between 10 and 25 frames per second depending on how much image manipulation features we added and which image resolution we used.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/QuQ4vf5Q6md2wAl6iZgJ"><img loading="lazy" width="1000" height="562" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/QuQ4vf5Q6md2wAl6iZgJ" alt="Coral USB Accelerator" /></a> </p><p>The USB accelerator is connected with the Raspberry Pi 4 via the USB 3.0 Type C interface. While the accelerator also supports USB 2.0, it is recommended to use USB 3.0 to ensure sufficient data transfer rates. The Pi 4 is the first Pi which has USB 3.0 on board. You can also use older Raspberry Pi versions but expect USB 2.0 to be a bottleneck which will substantially lower the achievable framerate. <a href="https://www.youtube.com/watch?v=cP2l3NUwlHQ&feature=youtu.be" target="_blank">Have a look</a>  at the framerate from our experiment we did two years ago with a Pi 3 and the Movidius stick, which was connected via USB 2.0.</p><p>We used a Logitech C920 HD Pro webcam for the setup but as we mentioned earlier, many webcams should work and should lead to similar results. After connecting the devices as shown below we can start to install the needed drivers and libraries.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/U0l76zSeSq2bhhB5UkGz"><img loading="lazy" width="700" height="525" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/U0l76zSeSq2bhhB5UkGz" alt="Hardware setup" /></a> </p><h2>Setup and installation</h2><p>First you should install a clean Raspbian distribution on your Raspberry Pi using the Noobs installer. A detailed guide can be found <a href="https://magpi.raspberrypi.org/articles/set-up-raspberry-pi-4" target="_blank">here</a> . With your Raspberry Pi up and running you can install git and clone the <a href="https://gitlab.codecentric.de/christoph.knauf/face_replace_demo" target="_blank">repository</a>  we prepared to help you get started fast. Installing all needed dependencies for the USB accelerator is quite some work. To save you the time and effort we wrote a script which automates the installation. You can find it in the root folder of the repository. Simply run <code>install.sh</code>, which is located in the root folder and it will install all dependencies. The installation will take a while. After the installation unplug and replug the Coral USB Accelerator once. Now you should be able to run the face replace demo with <code>python3.7 -m face_replace</code>.</p><h2>Face detection</h2><p>First we have to initialize the <a href="https://coral.withgoogle.com/docs/reference/edgetpu.detection.engine/#edgetpu.detection.engine.DetectionEngine" target="_blank">detection engine</a>  with the pre-trained model contained in the repository.

```python
from edgetpu.detection.engine import DetectionEngine

face_detection_engine = DetectionEngine(FACE_DETECTION_MODEL_PATH)
```

Now we can read the video stream from the webcam. Therefore, we are using the <a href="http://github.com/jrosebr1/imutils" target="_blank">image-utils</a>  library. To allow the camera sensor to warm up we wait 1 second before we start processing the stream.

```python
from imutils.video import VideoStream

video_stream = VideoStream(src=0).start()
time.sleep(1.0)
```

The next step is to read the single frames from the video stream and preprocess them. Since the video is recorded mirrored, we flip each frame horizontally using the computer vision library <a href="https://opencv.org/" target="_blank">opencv</a> . This code, together with the rest of the frame processing, happens inside a loop which runs infinitely until the user stops the application.

```python
import cv2

while True:
    input_frame = cv2.flip(video_stream.read(), 1) 
    frame_as_image, resized_frame = frame_processor.preprocess(input_frame)

```

As the first step of the image preprocessing, we resize the frame. The frame we captured from the video is a numpy ndarray. The color model of the pixel stored in this array is BGR. Since we need an RGB image as input for the face detection engine, we have to convert the colors from BGR to RGB and create an image out of the array using the imaging library <a href="https://pypi.org/project/Pillow/" target="_blank">Pillow</a>  (PIL).

```python
import imutils
from PIL import Image

def preprocess(frame):
    resized_frame = imutils.resize(frame, width=IMAGE_WIDTH)
    rgb_array = cv2.cvtColor(resized_frame, cv2.COLOR_BGR2RGB)
    frame_as_image = Image.fromarray(rgb_array)
    return frame_as_image, resized_frame
```

Using the preprocessed image and the previously initialized model we can now start to run the face detection. Therefore, we call the method <code>detect_with_image</code> on the previously initialized model. This will run the inference, which means it will produce the predicted faces. The method takes multiple inputs: the image, a <code>threshold</code> which defines the minimum confidence for the detected faces and the <code>top_k</code> parameter, which defines the maximum number of faces the model should detect.

```python
face_detection_engine.detect_with_image(
    frame_as_image,
    threshold=confidence,
    keep_aspect_ratio=True,
    relative_coord=False,
    top_k=MAX_FACES,
)
```

The detected faces are a list of <a href="https://coral.withgoogle.com/docs/reference/edgetpu.detection.engine/#edgetpu.detection.engine.DetectionCandidate" target="_blank"><code>DetectionCandidates</code></a>  where each entry provides the bounding box of the detected face.</p><h2>Face overlay</h2><p>In the next step we iterate over the detected faces, extract each bounding box and use it to overlay the faces with a face filter. How we determine the face filter will be explained in the next chapter. Notice, we apply the face filter on the resized frame (ndarray) instead of the Pillow image.

```python
for face in detected_faces:
    bounding_box = face.bounding_box.flatten().astype("int")
    face_filter = cache.update(bounding_box)
    frame = frame_processor.replace_face(
        bounding_box, resized_frame , face_filter
    )
```

We achieve this by extracting the coordinates from the bounding box and resizing the face filter to the size of the bounding box.

```python
(bbox_x1, bbox_y1, bbox_x2, bbox_y2) = bounding_box
width = bbox_x2 - bbox_x1
height = bbox_y2 - bbox_y1
face_filter_resized = cv2.resize(
    face_filter, (width, height), interpolation=cv2.INTER_AREA
)
```

Afterwards, we can overwrite the original face with the resized face filter. We rewrite all pixels inside the bounding box. Since our face filters are PNG images and want to keep the transparent regions of the images we have to take the alpha value into account and draw the original image with the inverted alpha value of the face filter image.

```python
face_filter_alpha = face_filter[:, :, 3] / 255.0
inverted_alpha = 1.0 - face_filter_alpha
for colour_index in range(0, 3):
    frame[bbox_y1:bbox_y2, bbox_x1:bbox_x2, colour_index] = (
        face_filter_alpha * face_filter[:, :, colour_index]
        + inverted_alpha * frame[bbox_y1:bbox_y2, bbox_x1:bbox_x2, colour_index]
    )
```

The last step is to display the manipulated frame. We can easily do this by calling the method <code>imshow</code> and providing the window name and the frame inside the constructor.

```python
cv2.imshow(window_name, frame)
```

<h2>Face tracking</h2><p>Until now, we didn’t explain how we keep track of faces and cover a face with the same face filter over time. Let’s say you would choose a filter randomly. Since the application does not yet know any time dependency, different face filters would be randomly chosen for the same person for every frame, which would result in a very chaotic filter flickering. Instead, our goal should be to track faces. That’s why we implemented a simple tracking algorithm which assigns a specific face filter to each face even when the position of the face changes from frame to frame.</p><h3>Caching – keeping faces in memory</h3><p>If you already worked with face detection the first idea which might come to your mind is to apply feature recognition to each detected face and compare the features from frame to frame, which would allow you to track the face. On the one hand this approach would be very expensive computationally speaking, on the other hand it would also be complex to implement.</p><p>Instead, we came up with the idea to store the bounding boxes of the detected faces, together with the related face filter, inside a cache. Using this cache, we calculate the nearest bounding box from frame to frame in order to rediscover the face related to the bounding box. This approach is much easier to implement and requires significantly fewer calculations. Though, it leads to an unwanted feature. When person B walks in front of person A, they might be able to “steal” the face filter from person A, since person A’s face won’t be visible while covered by person B and the closest face to A’s filter will then be the face of person B. We did not mind this feature for our experiments ;-).</p><p>Each entry of the cache contains the bounding box of the detected face, a face filter randomly chosen from the available face filter collection and the age of the entry

```python
class Cache:
   def __init__(self, face_filters):
        self.entries = []
        self.available_face_filters = face_filters
```

For each detected face we update the cache with its bounding box and get the face filter for the face.

```python
def update(self, bbox):
    if len(self.entries) == 0:
        return self._add_new_entry(bbox)
    nearest_bbox_distance, nearest_bbox_index = self._nearest_bounding_box(bbox)
    if nearest_bbox_distance > Cache.MAX_BBOX_DISTANCE:
        return self._add_new_entry(bbox)
    return self._update_entry(bbox, nearest_bbox_index)
```


If the cache is empty or the distance of the new bounding box to the cached bounding box exceeds a defined threshold, we add a new entry to the cache and return a new randomly chosen face filter.

```python
def _add_new_entry(self, bbox):
    face_filter = random.choice(self.available_face_filters)
    self.entries.append([bbox, face_filter, Cache.INITIAL_AGE])
    return face_filter

```

Otherwise, we update the cache entry with the nearest bounding box. This means we overwrite the bounding box of the cached entry with the new one, reduce the age of the entry and return the previously applied face filter.

```python
def _update_cache_entry(self, bbox, nearest_bbox_index):
    self.entries[nearest_bbox_index][0] = bbox
    self.entries[nearest_bbox_index][2] -= Cache.REJUVENATE
    return self.entries[nearest_bbox_index][1]
```

To find the nearest bounding box we perform a nearest neighbor lookup using the <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.cKDTree.html" target="_blank">k dimensional search tree</a>  provided by the <a href="https://www.scipy.org/" target="_blank">scipy library</a> .

```python
from scipy.spatial import cKDTree

def _nearest_bounding_box(self, bbox):
    bb_matrix = [entry[0] for entry in self.entries]
    nearest_bbox_distance, nearest_bbox_index = cKDTree(bb_matrix).query(bbox, k=1)
    return nearest_bbox_distance, nearest_bbox_index
```

<h3>Cache invalidation</h3><p>Our caching approach allows us to track faces but when the application is running, the cache will grow and use up an increasing amount of memory. Furthermore, it will contain bounding boxes of faces which left the captured area of the camera. For these reasons we decided to invalidate the cache each 10 iterations of the video loop.

```python
if num_iterations % 10 == 0:
    cache.invalidate()
```

The invalidate method first increases the age of each cache entry and then drops all entries whose age is equal or bigger than the maximum age.

```python
def invalidate(self):
     aged_entries = [
         [entry[0], entry[1], entry[2] + Cache.AGING] for entry in self.entries
     ]
     self.entries = [entry for entry in aged_entries if entry[2] < Cache.MAX_AGE]
```

<h2>Your idea?</h2><p>In this blog post we showed how you can build your own AI based face detection application using low cost consumer hardware and little machine learning knowledge.<br /> Now that you know how it works, you could try to build your own applications. For example, have a look at the <a href="https://www.youtube.com/watch?v=ruGk99s9Yhk" target="_blank">pong game</a> .</p><p>We hope that we inspired you to start your own experiments. Share your ideas or results in the comments below and let us know which experiments we should conduct next!</p><p>We thank our colleague Marcel Mikl for his support during the implementation of the demo.</p>

Realtime face detection and filtering with the Coral USB accelerator

<p>In this article series, we learn how to generate and publish HTML architecture documentation from code with Structurizr and Asciidoctor.</p><p>The goal of this approach is to reduce the efforts for maintaining long-living architecture documentation, keep it up to date, and ensure consistency.</p><p>In this first part of the series we will learn about the workflow behind this approach, the main tools used, and their core concepts.</p><h2>Motivation</h2><blockquote><p><em>Teams with good documentation deliver software faster and more reliably</em><br /> <em>than those with poor documentation </em><br /> <a href="https://cloud.google.com/blog/products/devops-sre/announcing-dora-2021-accelerate-state-of-devops-report" target="_blank">Accelerate State of DevOps Report, Google 2021</a> </p></blockquote><p>As a consultant, I work with self-organized development teams where architecture documentation is one of the many team responsibilities. Maintaining a consistent and up-to-date architecture documentation can be a challenging and time-consuming task, especially when things change fast and the tooling isn’t appropriate. Disruptive workflows, missing automation and versioning as well as storing the documentation separately from the code are problems often encountered. As a result, architecture documents can become inconsistent and outdated.</p><p>This article series addresses these challenges, aiming to enable development teams to create their architecture documentation using the tool they know and like, their IDE. We will combine open-source tools to create a workflow in order to generate architecture documentation from code. The resulting documents will be published to GitHub Pages and Confluence automatically via GitHub Actions.</p><p>Let’s start with a high-level view on this workflow.</p><h2>The workflow</h2><p>At first glance, the workflow shown in the following figure might look complex, but most of the steps are automated (gear icon). This automation has two benefits. First, we can easily adopt and republish the documentation if something changes. Second, we can focus on creating our architecture diagrams and writing the textual description of our system.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/uEHASwnvTTyzep1WZ8Sd"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/uEHASwnvTTyzep1WZ8Sd" alt="architecture documentation workflow" width="4181" height="2152" /></a> <p>The workflow we will implement in this series.</p></div><p>We create our architecture diagrams using Structurizr. These diagrams and the textual documentation will be combined in an AsciiDoc document, which will then be converted to HTML using the Asciidoctor tool chain.</p><p>The resulting document can then be published in company wikis or hosted on any website. As an example, we will publish them to GitHub Pages as well as to Atlassian Confluence, allowing non-technical colleagues to access the documentation.</p><p>The tools Structurizr and Asciidoctor as well as the markdown language AsciiDoc are at the heart of our workflow. Therefore, let’s have a closer look at them and their underlying concepts.</p><h2>Structurizr</h2><p>Structurizr is a collection of tools allowing us to create architecture diagrams programmatically. The approach of creating diagrams using code is not new. Tools like <a href="https://plantuml.com/de/" target="_blank">PlantUML</a>  or <a href="https://mermaid-js.github.io/mermaid/#/" target="_blank">Mermaid</a>  have been around for some years, allowing for writing diagrams in a markup or programming language that can be processed and converted to images. As a result, this enables us to treat our diagrams as code. Thus, we can store them in our version control system together with our code, edit them in our IDE, and use our build pipelines to generate them.</p><h3>Diagrams as code 2.0</h3><p>Taking this approach one step further, Structurizr introduces a single, central model representing the architecture of our system. For this model, we can create multiple views in order to visualize different aspects of our software on different abstraction levels. This separation between model and views has a big benefit. For example, let’s say we would use a diagram as code or drawing tool of our choice to create multiple architecture diagrams. Consequently, if an important aspect of our architecture changes, we would need to align all affected diagrams. This can be time-consuming and error-prone. With Structurizr, we only need to change the central model (<a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself" target="_blank">DRY principle</a> ). All views based on the model will be updated automatically, making sure they are consistent. To emphasize this advantage compared to other diagrams as code tools, the approach is referred to as <a href="https://dev.to/simonbrown/diagrams-as-code-2-0-82k" target="_blank">diagrams as code 2.0</a>  by the creator of Structurizr, <a href="https://twitter.com/simonbrown" target="_blank">Simon Brown</a> .</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/IakaS4uVT6eNGmQDIGcQ"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/IakaS4uVT6eNGmQDIGcQ" alt="docs as code 2.0 " width="1024" height="369" /></a> <p>Diagrams as code 2.0: Generating multiple diagrams from a central model, keeping them in sync automatically. Credits: Simon Brown</p></div><h3>Diagramming approach and the C4 model</h3><p>The creator of Structurizr is also the creator of the diagramming approach behind Structurizr, the <a href="https://c4model.com/" target="_blank">C4 model</a> . The goal of the C4 model is to provide an easy-to-learn approach to software architecture diagramming. At its core, the C4 model defines four hierarchical abstraction (or zoom) levels to describe the static structure of software. These abstraction levels are systems, containers, components, and code elements.</p><div><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/1ulERAojRcm0oKH5yOox" alt="architecture documentation: c4 model" width="1024" height="612" /> <p>The abstraction levels of the C4 model.</p></div><p>For those abstraction levels, the C4 model defines diagram types showing the software system in scope at different levels of detail. These diagram types are <a href="https://c4model.com/#SystemContextDiagram" target="_blank">the system context</a> , <a href="https://c4model.com/#ContainerDiagram" target="_blank">container</a> , <a href="https://c4model.com/#ComponentDiagram" target="_blank">component</a> , and <a href="https://c4model.com/#CodeDiagram" target="_blank">code</a> diagram, while code level diagrams are not defined in detail by the C4 model. Instead, UML, ERM or similar diagrams should be used. As the following figure shows, each lower abstraction level diagram can be seen as a zoom-in view of the next higher abstraction level.</p><div><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/cgK0bEYRFWSjUo9XAHQw" alt width="1024" height="756" /> <p>The 4 core diagram types defined by the C4 model. Copyright Simon Brown. www.c4model.com</p></div><p>Structurizr allows for the definition of views for each of these diagram types except the code-level diagrams. It’s recommended to automatically generate the latter from source code (e.g. using our IDE).</p><p>Apart from the core diagrams, the C4 model defines supplementary diagram types. These are <a href="https://c4model.com/#DeploymentDiagram" target="_blank">deployment</a> , <a href="https://c4model.com/#DynamicDiagram" target="_blank">dynamic (interaction)</a>  and <a href="https://c4model.com/#SystemLandscapeDiagram" target="_blank">system landscape</a> diagrams, which are also supported by Structurizr. Describing all these diagram types and the related Structurizr model elements would go beyond the scope of this article. Instead, we will get to know the Deployment diagram during <a href="https://blog.codecentric.de/architecture-documentation-as-code-with-structurizr-and-asciidoctor-part-3-structurizr" target="_blank">part three</a> of this series.</p><h3>APIs, output formats, and rendering tools</h3><p>In order to create the model and views, Structurizr supports multiple languages and APIs. A textual <a href="https://structurizr.com/dsl" target="_blank">domain-specific language</a>  (DSL) as well as APIs for multiple programming languages can be used. In this article series, we will use the <a href="https://github.com/structurizr/java" target="_blank">Java API</a> . Compared to the DSL, it gives us more freedom regarding the choice of development tooling. Furthermore, it allows for automating the model creation, for example, via code analysis using reflection or parsing configuration files.</p><p>Structurizr supports multiple output formats like <a href="https://plantuml.com" target="_blank">PlantUML</a> , <a href="https://github.com/plantuml-stdlib/C4-PlantUML" target="_blank">C4-PlantUML</a>  or <a href="https://mermaid-js.github.io/mermaid/" target="_blank">Mermaid</a>  as well as a <a href="https://structurizr.com/products#comparison" target="_blank">web-based rendering and editing tool</a> . We will use C4-PlantUML as output format, since it fits well with the C4 model of Structurizr, it is open-source, and it allows for the creation of diagrams in a high visual quality which can easily be integrated in AsciiDoc documents.</p><p>However, there is a gap between the features provided by Structurizr and the capabilities of C4-PlantUML. In order to bridge this gap, I wrote a <a href="https://github.com/chriskn/structurizr-c4puml-extension" target="_blank">Kotlin-based library</a>  on top of the Java API which extends the Structurizr model and provides extended C4-PlantUML support. Furthermore, the library provides a more Kotlin-like syntax and allows us to modify the layout of the resulting C4-PlantUML diagram, which is crucial when diagrams get more complex. Therefore, we will use this library in this series. The following figure compares the rendering tools mentioned in this chapter.</p><div><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/b0uLaJUKQP2klQcEzism" alt="rendering tools" width="1024" height="942" /> <p>Example component diagram rendered with the web rendering tool (top left), C4-PlantUML (top centre), PlantUML (top right), Mermaid (bottom left) and the extension we will use (bottom right)</p></div><p>The tool ecosystem around Structurizr is constantly growing in terms of official tools, but also third-party libraries. At the time of this writing, <a href="https://github.com/search?q=Structurizr&type=repositories" target="_blank">a search on GitHub for “Structurizr”</a>  results in 170 repositories. So, if the combination of tools used in this article doesn’t suit your needs, it’s likely that there is an adaptation out there that fits for you.</p><h2>AsciiDoc</h2><p>AsciiDoc is a plain text markup language built for writing technical documents. The AsciiDoc syntax is lightweight and human-readable. It provides all the basic formatting features (text formatting, lists, tables, …) you would expect from a markup language. Furthermore, AsciiDoc provides features for writing technical documents like syntax highlighting for code blocks, document and image includes, cross references and many more out of the box. The figure below shows an example AsciiDoc file and the generated HTML.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/l1y9FFb5RVGvilofJkqO"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/l1y9FFb5RVGvilofJkqO" alt="asciidoc file with HTML" width="1024" height="527" /></a> <p>AsciiDoc document and the generated HTML output</p></div><p>If additional features are needed, AsciiDoc can be <a href="https://asciidoctor.org/docs/extensions/" target="_blank">extended</a> . This extension mechanism is a core aspect of AsciiDoc and allows for adding new syntax elements, macros, and document post-processors.</p><h3>Asciidoctor</h3><p>AsciiDoc itself is not coupled to any specific output format. Instead, AsciiDoc documents are parsed and then converted to HTML, man-pages, PDF, or any other output format. This is done by an AsciiDoc processor. The reference AsciiDoc processor implementation is called Asciidoctor.  Asciidoctor is open-source and was originally written in Ruby. Now there is a JVM adaptation called AsciidoctorJ and a JavaScript adaptation called Asciidoctor.js. In this article series, we will use AsciidoctorJ.</p><p>AsciidoctorJ provides, among other tools, the <a href="https://github.com/asciidoctor/asciidoctor-gradle-plugin" target="_blank">Gradle plugin</a>  we will use to generate HTML documents from our AsciiDoc files. Furthermore, we will use an <a href="https://github.com/asciidoctor/asciidoctorj-diagram" target="_blank">extension</a>  to integrate the C4-PlantUML diagrams generated with Structurizr and convert them to SVG images. Besides Asciidoctor, there is a lot more of tooling around AsciiDoc. For example, WYSIWYG (What You See Is What You Get) editor plugins for the most common IDEs like <a href="https://marketplace.visualstudio.com/items?itemName=asciidoctor.asciidoctor-vscode" target="_blank">VS Code</a> , <a href="https://marketplace.eclipse.org/content/asciidoctor-editor" target="_blank">Eclipse,</a>  or <a href="https://plugins.jetbrains.com/plugin/7391-asciidoc" target="_blank">IntelliJ</a> .</p><p>In summary, the plain text syntax of AsciiDoc together with the automation provided by Asciidoctor and the IDE plugins allow us to define our documents as code. As for our diagrams, this means we can store AsciiDoc files in our version control system together with our code, edit them in our IDE, and use our build tools to generate and publish them automatically.</p><h2>Summary and outlook</h2><p>Maintaining a consistent and up-to-date architecture documentation can be a challenging and time-consuming task. Disruptive workflows, missing automation and versioning as well as storing the documentation separately from the code can lead to inconsistent and outdated documents. In this article, a workflow based on open-source tools was introduced, aiming to address these challenges.</p><p>The core of this workflow are Structurizr and Asciidoctor. Both follow the “as code” approach, allowing developers to maintain their architecture documentation with the tools they are already familiar with. Combining them in an automated tool chain enables development teams to write maintainable, long-living architecture documents.</p><p>In the following articels, we will implement the workflow described in this article. Starting with <a href="https://blog.codecentric.de/architecture-documentation-as-code-with-structurizr-and-asciidoctor-part-2-asciidoctor" target="_blank">part two</a>, where we will use AsciiDoc to write architecture documentation based on the arc42 template and following the documentation as code approach. Furthermore, we will automate the generation of HTML documents out of the AsciiDoc files using Asciidoctor and Gradle.</p>

Architecture docs as code with Structurizr & Asciidoctor. Part 1: Workflow and tooling

You are reading the fourth part of this article series about architecture documentation as code.  If you worked through the previous articles, you already automated the generation of your architecture documents using <a href="https://blog.codecentric.de/architecture-documentation-as-code-with-structurizr-and-asciidoctor-part-2-asciidoctor" target="_blank">Asciidoctor</a> and integrated the diagrams generated with <a href="https://blog.codecentric.de/architecture-documentation-as-code-with-structurizr-and-asciidoctor-part-3-structurizr" target="_blank">Structurizr</a>. 

Now, let's publish the resulting documentation. The related part of our workflow is displayed on a gray background in the following figure.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Ha5ut9dlRwGv8I9nmlKn" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Ha5ut9dlRwGv8I9nmlKn" width="5574" height="2869" />
		</a>
		<p>The workflow we will implement in this article. See  <a href="https://blog.codecentric.de/architecture-documentation-docs-as-code-structurizr-asciidoctor">part one</a> of this series.</p>
	</div>
<p>
</center>

The code can be found on <a href="https://github.com/chriskn/arch-docs-as-code-example" target="_blank">GitHub</a>. The architecture documentation we create in this article can be found on <a href="https://chriskn.github.io/arch-docs-as-code-example/" target="_blank">GitHub Pages</a>.

## Publishing pipeline

Like for our code, we will build and publish the documentation continuously and in an automated way as part of our build pipeline. Therefore, we will define two build jobs using GitHub Actions. One build job will publish the document to GitHub Pages, the other one will publish it to Atlassian Confluence.
Therefore, we create a file named `docs-pipeline.yml` under `.github/workflows`.

## GitHub Pages
Before we start defining the pipeline, we need to tell GitHub that we want to use GitHub Actions as deployment method for GitHub Pages. Therefore, we open the settings of our repository, navigate to the Pages section and choose GitHub Actions as source for build and deployment.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/LqO1yoBQYmxvT7dwpD1T" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/LqO1yoBQYmxvT7dwpD1T" width="1245" height="271" />
		</a>
		<p>Setting GitHub Actions as source for GitHub Pages.</p>
	</div>
<p>
</center>

Afterwards, we can configure the build job as shown in the following listing, running each time a developer pushes to the main branch.

```yaml
name: Docs pipeline

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write

# Allow one concurrent deployment
concurrency:
  group: "docs_pages"
  cancel-in-progress: true

on:
  push:
    branches: [main]

jobs:
  docs_pages:
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v3
      # We need to install GraphViz to convert PlantUML files to images
      - name: Setup Graphviz
        uses: ts-graphviz/setup-graphviz@v1
      # Run Asciidoctor
      - name: Asciidoctor
        uses: gradle/gradle-build-action@v2
        with:
          arguments: asciidoctor
      # Upload the HTML docs generated with Asciidoctor
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: ./build/docs
      # Deploy the uploaded docs using GitHub Pages action
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v1
```

To publish GitHub Pages, we need to set `permissions` for the GitHub token. Furthermore, we allow only one concurrent page deployment to avoid race conditions when multiple pipelines are running in parallel. In such cases, the job started earlier will be cancelled.

To be able to deploy to GitHub Pages, we need to use the GitHub Pages `environment` for our job. Furthermore, we have to install Graphviz for the PlantUML to SVG conversions. Therefore, we use a predefined action. 

Afterwards, we run the Asciidoctor Gradle task and upload the resulting HTML documents using the `upload-pages-artifact` action. Finally, we deploy the uploaded artefacts using the `deploy-pages action`. 

After pushing the workflow file to the main branch, it will be executed. When the workflow is done, you should see the following result in the workflow view.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qdSE99pGTmaTTRPSkGzU" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qdSE99pGTmaTTRPSkGzU" width="530" height="184" />
		</a>
		<p>The GitHub workflow view after the docs_pages job was finished.</p>
	</div>
<p>
</center>

A click on the link will open the architecture documentation hosted on GitHub Pages.

### Atlassian Confluence
Some companies I worked for had the policy that all documentation has to be published in Confluence. Let's fulfil this policy and allow non-technical reader to access the documentation in Confluence. 

Therefore, we use the official  <a href="https://github.com/confluence-publisher/confluence-publisher" target="_blank">Confluence Publisher</a> provided by Atlassian. The publisher will convert our AsciiDoc files to Confluence-compatible XHTML and will upload the resulting pages using the Confluence REST API.

Since the publisher is only available as a Maven plugin, we can't directly use it. Instead, we will use a <a href="https://github.com/nomisp/confluence-publisher-plugin" target="_blank">Gradle plugin</a> wrapping the Confluence Publisher. Therefore, we add the plugin to the `plugins` section of the `build.gradle.kts` file.

```kotlin
plugins {
    ...
    id("ch.nomisp.confluence.publisher") version "0.2.0"
}
```

Afterwards, we configure the publisher as shown in the following listing.

```kotlin
confluencePublisher {
    asciiDocRootFolder.set(tasks.asciidoctor.get().sourceDir)
    setAttributes(tasks.asciidoctor.get().attributes)
    rootConfluenceUrl.set("YOUR_CONFLUENCE_URL")
    spaceKey.set("YOUR_SPACE_KEY")
    ancestorId.set("ID_OF_PARENT_PAGE")
    // set username or password or use an api token as password with empty username
    username.set("")
    password.set(System.getenv("CONFLUENCE_TOKEN"))
}
```

To make sure the output of the Confluence Publisher task is the same as the one produced by the Asciidoctor task, we reuse the `sourceDir` and the `attributes` configured in the Asciidoctor task. The attributes `confluenceUrl`, `spaceKey` and `ancestorId` are needed to tell the publisher where to upload the documentation. The `ancestorId` is the ID of the parent page of our documentation. It’s a good idea to create a dedicated parent page for the architecture documentation. For the authentication against Confluence, we can set `username` and `password`. However, using a Confluence API token is recommended. To do so, the `username` has to be empty, and the token must be used as `password`. More properties can be configured. The complete list can be found in the <a href="https://confluence-publisher.atlassian.net/wiki/spaces/CPD/overview?homepageId=315719687" target="_blank">documentation</a>.

To make sure the diagrams included in the documentation are up to date, we let the `PublishToConfluenceTask` depend on the `writeDiagrams` task.

```kotlin
tasks.withType(PublishToConfluenceTask::class) {
    dependsOn("writeDiagrams")
}
```

The Confluence Publisher by convention creates a Confluence page for each AsciiDoc file which does not start with an underscore. This means our documentation would end up in Confluence as one combined file based on the `index.adoc` and additionally a separate file for each included section file. To avoid this, we have to rename the included section files and let each file name start with an underscore. As a result, only one Confluence page based on our `index.adoc` file is created.

> **Note**: Another convention is that each AsciiDoc file to be uploaded as a Confluence page must contain a top-level document title (e.g. "= Some title"). This title will be used as the Confluence page title. Due to limitations of the Confluence REST API, these titles must be unique per Confluence space. Therefore, if you want to upload multiple pages with the same title in the same space, you must pre- or suffix them. This can be done using the `pageTitlePrefix` and `pageTitleSuffix` attributes provided by the Confluence publisher.

To automate the Confluence publishing, we add the following job to the `docs-pipeline.yml` file.

```yaml
docs_confluence:
  runs-on: ubuntu-latest
  steps:
    - name: Checkout Repo
      uses: actions/checkout@v3
    # We need to install GraphViz to convert PlantUML files to images
    - name: Setup Graphviz
      uses: ts-graphviz/setup-graphviz@v1
    # Run publishToConfluence task which runs Asciidoctor 
    # and publishes the resulting file to Confluence
    - name: Publish To Confluence
      uses: gradle/gradle-build-action@v2
      # Add the Confluence token provided as Action secret
      # to the environment 
      env:
        CONFLUENCE_TOKEN: ${{secrets.CONFLUENCE_TOKEN}}
      with:
        arguments: publishToConfluence
```

Again, we check out the repository and install Graphviz. Afterwards, we run the `publishToConfluence` Gradle task. The Confluence API token needed by the task must be part of the step environment. It can be stored as action secret in the security settings of the repository as shown in the following figure. The <a href="https://github.com/chriskn/arch-docs-as-code-example/blob/161f3a8332a42aba222325964005686981c10f04/.github/workflows/docs-pipeline.yml" target="_blank">complete workflow</a> can be found in the example repository. 

<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ecRRoYcQRLqhnx5iunJk" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ecRRoYcQRLqhnx5iunJk" width="530" height="184" />
		</a>
		<p>Configuring an repository secret for GitHub Actions.</p>
	</div>
<p>
</center>

The resulting Confluence page can be found in the Confluence space you configured beneath the page you configured as ancestor. The following figure shows an excerpt of the Confluence page produced by the example project. 
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/80AZil7SXmKAS2WTYG8A" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/80AZil7SXmKAS2WTYG8A" width="144" height="513" />
		</a>
		<p>Excerpt of the Confluence page produced by the example project.</p>
	</div>
<p>
</center>

## Summary and outlook
In this article we finished the implementation of our architecture documents as code workflow.  We learned how to publish our architecture documentation automatically to GitHub Pages and Atlassian Confluence as part of a build pipeline using GitHub Actions. 

As described in part one, the goal of this workflow is to reduce the efforts for maintaining long-living architecture documentation, keep it up to date, and ensure consistency. We already reduced the maintenance efforts by automating document generation and deployment. Furthermore, Structurizr with its central model saves us maintenance efforts when changing our diagrams and ensures that our diagrams are consistent.

The upcoming <a href="https://blog.codecentric.de/architecture-docs-as-code-structurizr-asciidoctor-part-5-generating-documentation" target="_blank">final part</a> of this series is about further automating the workflow by extracting Structurizr model elements from application code, configurations and API descriptions. 

Architecture docs as code with Structurizr & Asciidoctor. Part 4: Publishing

You are reading the second part of this article series about architecture documentation as code. In this article, we will implement the Asciidoctor-related part of the workflow highlighted in the following figure.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/LiiizD7oQsuRQD877mJK" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/LiiizD7oQsuRQD877mJK" width="5574" height="2869" />
		</a>
		<p>The workflow we will implement in this article series.</p>
	</div>
<p>
</center>

We will use AsciiDoc to write architecture documentation based on the arc42 template and automate the generation of HTML documents out of these files using Asciidoctor and Gradle. Furthermore, we prepare the integration of the C4-PlantUML diagrams, which we will create using Structurizr in <a href="https://blog.codecentric.de/architecture-documentation-as-code-with-structurizr-and-asciidoctor-part-3-structurizr" target="_blank">part three</a> of this series. 

The motivation behind this workflow, the main tools used, and their core concepts are explained in <a href="https://blog.codecentric.de/architecture-documentation-docs-as-code-structurizr-asciidoctor" target="_blank">part one</a> of this series.

The code can be found on <a href="https://github.com/chriskn/arch-docs-as-code-example" target="_blank">GitHub</a>. The architecture documentation we create in this article series can be found on <a href="https://chriskn.github.io/arch-docs-as-code-example/" target="_blank">GitHub Pages</a>.

## Project setup
When treating our documentation like code, it makes sense to store it together with the code it documents. As a result, we can integrate the documentation in review processes, keeping documentation and code in sync more easily and build the documentation together with our application. Furthermore, this allows us to generate parts of the documentation automatically from our application code and configuration, which will be covered in one of the next parts of this series.

For more complex projects than the one used in this article, techniques like multi-module projects, Git-Submodules or shared documentation libraries could be used. For the simple example used in this article, it's sufficient to store the documentation code in a separate source-set next to the test and implementation code.  

Therefore, we create a new source-set called `docs` by adding the following lines to the `build.gradle.kts` file.

```kotlin
sourceSets {
    create("docs") {
        kotlin {
            compileClasspath += main.get().output
            runtimeClasspath += output + compileClasspath
        }
    }
}
```

To use the compiled application code, as well as related dependencies and configurations, we add the main source-set output path to the compile and runtime classpath of the `docs` source-set. As a result, the src folder should have the following structure.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/IwjXl3y5QF2Gl06JUSeN" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/IwjXl3y5QF2Gl06JUSeN" width="200" height="166" />
		</a>
		<p>The source folders of the example project.</p>
	</div>
<p>
</center>

## Documentation
We will write our documentation using AsciiDoc. To edit AsciiDoc files like our code, we should first install a WYSIWYG (What You See Is What You Get) editor plugin for our IDE. Fortunately, there are plugins available for most IDEs. For example, <a href="https://marketplace.visualstudio.com/items?itemName=asciidoctor.asciidoctor-vscode" target="_blank">VS Code</a>, <a href="https://marketplace.eclipse.org/content/asciidoctor-editor" target="_blank">Eclipse</a>, or <a href="https://plugins.jetbrains.com/plugin/7391-asciidoc" target="_blank">IntelliJ</a>. After the installation, we can edit AsciiDoc documents using features like syntax highlighting, auto-completion and a live preview. The following figure shows the editor provided by the IntelliJ plugin.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/b37iVXPtSyC8KgsWwPIc" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/b37iVXPtSyC8KgsWwPIc" width="2674" height="1612" />
		</a>
		<p>The IntelliJ AsciiDoc plugin.</p>
	</div>
<p>
</center>

### What should be documented?
One of the main questions when writing architecture documentation is:

What should we document about our architecture?

This question is not easy to answer. While there are aspects of the architecture like quality requirements, architecture decisions, or the system context which are essential parts of an architecture documentation, other aspects might not be relevant. What is relevant or not depends on things like the system to be documented, external factors like legal requirements, and the documentation's target audience.

Another aspect to consider it the level of detail. Detailed information tends to outdate faster than more general information. As a result, detailed documents need to be updated more often, resulting in more efforts to keep the documentation up-to-date. Therefore, the goal is to find the right balance between what’s really relevant and the efforts needed to maintain the documentation.

However, architecture document templates can help with answering the question of what should be documented. Such templates describe the contents as well as the document structure of an architecture documentation. Multiple of those templates are freely available on the internet.

### The arc42 template
In this article, we will use the <a href="https://arc42.org/overview" target="_blank">arc42 template</a>. The document structure defined by this template is shown in the following figure.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/3KbXf4sWS0KNcFRYC6JM" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/3KbXf4sWS0KNcFRYC6JM" width="1911" height="802" />
		</a>
		<p>The chapters of the arc42 template. Source: www.arc42.org</p>
	</div>
<p>
</center>

Apart from the template itself, the arc42 website provides numerous useful <a href="https://docs.arc42.org/home" target="_blank">tips</a> and <a href="https://arc42.org/examples" target="_blank">examples</a> regarding how to use the template. Furthermore, it describes how the template can be tailored depending on whether you want to follow a <a href="https://docs.arc42.org/keywords/#lean" target="_blank">lean</a> or a <a href="https://docs.arc42.org/keywords/#thorough" target="_blank"> thorough</a> approach or if you only want to document the <a href="https://docs.arc42.org/keywords/#essential" target="_blank">essential</a> aspects of your software architecture.

<a href="https://arc42.org/download" target="_blank">Besides other formats and languages</a>, there is a ready to use AsciiDoc version of the template available in English. We will use this template. After <a href="https://github.com/arc42/arc42-template/raw/master/dist/arc42-template-EN-plain-asciidoc.zip" target="_blank">downloading</a> it, we copy all AsciiDoc files, including the ones in the src folder, to the `src/docs/resources` project folder. This flat folder structure will make it easier to convert the AsciiDoc files to HTML. Afterwards, the folder should have the structure shown in the following figure.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tGU3SLUXQVOJpp2QPIIs" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tGU3SLUXQVOJpp2QPIIs" width="349" height="426" />
		</a>
		<p>The structure of the src/docs/resources folder.</p>
	</div>
<p>
</center>

Each section of the template is reflected by an AsciiDoc file. Feel free to tailor the template to your needs by removing section files and the corresponding include statements in `arc42-template.adoc` file.
Afterwards, we rename the `arc42-template.adoc` to `index.adoc`, since this file will result in the index.html file of our generated HTML documentation.
Next, we open the `index.adoc` file and replace the following line with our own document title.

```asciidoc
= image:arc42-logo.png[arc42] Template
```

The following lines can also be removed, since the containing CSS only relate to the arc42 help we don’t use.

```asciidoc
ifdef::backend-html5[]
++++
<style>
#header,
#content,
#footnotes,
#footer{
    width:80%;
    margin-left:auto;
    margin-right:auto;
    margin-top:0;
    margin-bottom:0;
    max-width:70em;
    *zoom:1;
    position:relative;
    padding-left:.9375em;
    padding-right:.9375em
}
</style>
++++
endif::backend-html5[]
```

The following two lines can be removed as well, together with the related documents. That's because we don’t need an about arc42 section, and we will configure the attributes stored in the `config.adoc` file using Gradle and Asciidoctor later.

```asciidoc
include::src/config.adoc[]
 
include::src/about-arc42.adoc[]
```

Finally, we need to fix the remaining includes by removing the `src/` part of the path for each include statement.
As a result, we should end up with an `index.adoc` file containing only the title, table of contents and the section includes, as shown in the following listing.

```asciidoc
= Example documentation
 
// toc-title definition MUST follow document title without blank line!
:toc-title: Table of Contents
 
// numbering from here on
:numbered:
 
<<<<
// 1. Introduction and Goals
include::01_introduction_and_goals.adoc[]
 
<<<<
// 2. Architecture Constraints
include::02_architecture_constraints.adoc[]
 
<<<<
// 3. System Scope and Context
include::03_system_scope_and_context.adoc[]
 
<<<<
// 4. Solution Strategy
include::04_solution_strategy.adoc[]
 
<<<<
// 5. Building Block View
include::05_building_block_view.adoc[]
 
<<<<
// 6. Runtime View
include::06_runtime_view.adoc[]
 
<<<<
// 7. Deployment View
include::07_deployment_view.adoc[]
 
<<<<
// 8. Concepts
include::08_concepts.adoc[]
 
<<<<
// 9. Architecture Decisions
include::09_architecture_decisions.adoc[]
 
<<<<
// 10. Quality Requirements
include::10_quality_requirements.adoc[]
 
<<<<
// 11. Technical Risks
include::11_technical_risks.adoc[]
 
<<<<
// 12. Glossary
include::12_glossary.adoc[]

```

## Asciidoctor
Now that we have our AsciiDoc files in place, we can start to automate their conversion to HTML using Asciidoctor. Therefore, we add the Asciidoctor Gradle plugin to the plugins section of our `build.gradle.kts`.

```kotlin
plugins {
    ....
    id("org.asciidoctor.jvm.convert") version "3.3.2"
}
```

Afterwards, we configure the conversion using the AsciidoctorTask as shown in the following listing:

```Kotlin
tasks.withType(AsciidoctorTask::class) {
    setSourceDir(file("./src/docs/resources"))
    setBaseDir(file("./src/docs/resources"))
    setOutputDir(file("build/docs"))
    isLogDocuments = true
    options(mapOf("doctype" to "book"))
}
```

This is just a basic configuration. More information on how to configure the task can be found in the <a href="https://asciidoctor.github.io/asciidoctor-gradle-plugin/development-3.x/user-guide" target="_blank">gradle plugin documentation</a>. However, with this basic configuration, we should be able to convert our AsciiDoc files by running the Asciidoctor Gradle task:

```bash
gradle asciidoctor
```

As a result, the HTML output can be found in the `build/docs` folder as shown in the following figure.
<center>
</p>
	<div>
		<a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/REihNJE6SSAmGebVfFPk" target="_blank">
			<img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/REihNJE6SSAmGebVfFPk" width="386" height="402" />
		</a>
		<p>The resulting HTML files after executing the asciidoctor Gradle task.</p>
	</div>
<p>
</center>

### Configuration
Our AsciiDoc document is split into multiple files, which are included in the root `index.adoc` file. When working with multiple, included AsciiDoc files, there are usually some settings which have to be configured globally for all of these files. For example, the directory containing the C4-PlantUML diagrams we will include in the <a href="https://blog.codecentric.de/architecture-documentation-as-code-with-structurizr-and-asciidoctor-part-3-structurizr" target="_blank">next part</a>. Such settings a referred to as attributes by Asciidoctor.

One approach is to define those attributes in a dedicated AsciiDoc file and include this file in our root `index.adoc`. The arc42 template followed this approach with the `config.adoc` file we deleted in the section "The arc42 template". When the AsciiDoc files are converted to HTML, those attributes would be applied to the `index.adoc` and all included files by Asciidoctor.

While this approach works well for conversion, it does not work for the IDE plugin and its preview feature. The reason is that the IDE plugin does not recognize the attributes included in the `index.adoc` when previewing included files. As a result, the included files would be rendered differently in the IDE preview and the converted HTML document.

To overcome this limitation, we will generate an `.asciidoctorconfig` containing our attributes using the Asciidoctor editorconfig Gradle plugin. The generated configuration file will be recognized, and the contained attributes will be applied automatically to all AsciiDoc files by the IDE editor plugin.

Therefore, we define our attributes as a map in our `built.gradle.kts` as shown in the following listing.

```kotlin
val asciiAttributes = mapOf(
    "plantUmlDir" to "./plantuml/",
    "toc" to "left",
    "toclevels" to 3,
    "max-width" to "100%",
    "projectName" to rootProject.name,
    "buildTime" to SimpleDateFormat("dd-MM-yyyy HH:mm:ssZ").format(Date())
)
```

The listing demonstrates two additional benefits of this approach. Because we define the attributes in our `gradle.kotlin.kts`, we can use Gradle related variables like `rootProject.name`. Furthermore, we can compute the configuration values dynamically at build time, as shown with the `buildTime` attribute. 

To generate an `.asciidoctorconfig` containing these attributes, we add the editorconfig plugin to the plugins section of our `build.gradle.kts`.

```kotlin
plugins{
  ...
  id("org.asciidoctor.editorconfig") version "3.3.2"
}
```

Next, we configure the `AsciidoctorEditorConfigGenerator` task to use these attributes and set the destination for the `.asciidoctorconfig` file to the `src/docs/resources` folder.  

```kotlin
tasks.withType(AsciidoctorEditorConfigGenerator::class) {
    setAttributes(asciiAttributes)
    setDestinationDir("./src/docs/resources")
    group = "documentation"
}
```

To use these attributes also while converting the Asciidoctor files to HTML, we edit the Asciidoctor task and set the attributes.

```kotlin
tasks.withType(AsciidoctorTask::class)
  …
  attributes(asciiAttributes)    
}
```

Furthermore, to ensure that the `.asciidoctorconfig` is generated automatically, we let the `processDocsResources` task depend on the `asciidoctorEditorConfig` task.
 
```kotlin
tasks.named("processDocsResources"){
    dependsOn("asciidoctorEditorConfig")
}
```

### PlantUML conversion

We use <a href="https://docs.asciidoctor.org/diagram-extension/latest" target="_blank">Asciidoctor diagram</a> to convert the C4-PlantUML diagrams, which we will generate via Structurizr, to images. Asciidoctor diagram is a set of extensions that are already shipped with the `org.asciidoctor.jvm.convert` plugin. Such a set of extensions is referred to as module in Asciidoctor.

Since we already added the convert plugin, we only need to activate the diagram module by adding the following lines to our `build.gradle.kts` and set the version to the latest available.

```kotlin
asciidoctorj {
    modules {
        diagram.use()
        diagram.setVersion("2.2.1")
    }
}
```

The diagram module uses <a href="https://graphviz.org" target="_blank">Graphviz</a> to convert PlantUML (and other) diagrams to SVG, PNG or JPEG images. Therefore, we have to <a href="https://graphviz.org/download" target="_blank">download and install</a> Graphviz. Graphviz provides a program called dot, which is used to generate images from the textual diagram definitions. Detailed information regarding the installation of Graphviz for PlantUML can be found in the <a href="https://plantuml.com/en/graphviz-dot" target="_blank">PlantUML documentation</a>.

## Summary and outlook
We learned how to use AsciiDoc to write architecture documentation based on the arc42 template. We stored the documentation next to our code and used our IDE to edit it, following the documentation as code approach. Furthermore, we automated the generation of HTML documents out of the AsciiDoc files using Asciidoctor and Gradle.

In the <a href="https://blog.codecentric.de/architecture-documentation-as-code-with-structurizr-and-asciidoctor-part-3-structurizr" target="_blank">next part</a>, we use Structurizr to model our system, create C4-PlantUML diagrams from this model and integrate them in our documentation. 

Architecture docs as code with Structurizr & Asciidoctor. Part 2: Asciidoctor

Architecture docs as code with Structurizr & Asciidoctor. Part 3: Structurizr

Architecture docs as code with Structurizr & Asciidoctor. Part 5: Generating documentation

<h2>Introduction</h2><p>This article will give an introduction to Kotlin DSLs (Domain-Specific Languages) and teach you how to create and generate them with an annotation processor.<br /> We’ll also take a look at Kotlin internals and how we can use this knowledge to our advantage.</p><p>I’m only going to give a short introduction to DSLs here. For a more in-depth look at what (Kotlin) DSLs are, read <a href="/wissens-hub/blog/kotlin-dsl-apache-kafka">this article</a>  showcasing an Apache Kafka Kotlin DSL.<br /> With that said, let’s get into it:</p><h2>What is a DSL?</h2><p>DSLs are purpose-built languages used to describe objects of a certain domain. For example, HTML is a DSL to describe web pages.<br /> An easy way to define a DSL is to make it a subset of an existing language: HTML is a subset of XML, npm configuration files are a subset of JSON, and Gradle build files can be written with a subset of Groovy or Kotlin.</p><h2>Why Kotlin DSLs?</h2><p>While Gradle is the most well-known case of a Kotlin DSL for configuration, anybody can define their own DSL for any use case. For example, liquibase supports many configuration formats, <a href="https://github.com/F43nd1r/liquibase-kotlin-dsl" target="_blank">including Kotlin</a> .<br /> Kotlin DSLs have a major advantage over most alternatives: they’re essentially code, which means you can:</p><ul><li>write logic into your configuration (e.g. easily changing any part of your configuration based on environment)</li><li>get IDE (and compiler) support for syntax and type checking, formatting and autocompletion</li></ul><h2>Creating a Kotlin DSL</h2><p>Let’s take a look at a simple example:</p><div>

```kotlin
person {
    age = 24
    name = "Lukas Morawietz"
    hands = 2
}
```

</div><p>Looks nice, right?<br /> Here’s what we need to define such a DSL:</p><div>

```kotlin
data class Person(val age: Int, val name: String, val hands: Int)
 
data class PersonBuilder {
    var age: Int? = null
    var name: String? = null
    var hands: Int? = null
 
    fun build(): Person {
        checkNotNull(age) { "age must be assigned." }
        checkNotNull(name) { "name must be assigned." }
        checkNotNull(hands) { "hands must be assigned." }
        return Person(age, name, hands)
    }
}
 
fun person(initializer: PersonBuilder.() -> Unit): Person
    = PersonBuilder().apply(initializer).build()
```

</div><p>This part doesn’t look as nice anymore, basically everything past the first line is boilerplate code.</p><h2>Generating a Kotlin DSL</h2><p>Luckily, we don’t need to write all that boilerplate code, we can just write code to generate it for us – using annotation processing.<br /> We can use either <a href="https://github.com/google/ksp" target="_blank">ksp</a>  or <a href="https://kotlinlang.org/docs/kapt.html" target="_blank">kapt</a>  to generate code based on annotations and the <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree" target="_blank">Abstract Syntax Tree (AST)</a>  during compilation.<br /> In the end we want our code to look like this:</p><div>

```kotlin
@AutoDSL
data class Person(val age: Int, val name: String, val hands: Int)
```

</div><p>And have the processor generate all the boilerplate we had above for us.</p><h3>How do we do that?</h3><p>We just have a look at the constructor of our annotated class, and generate a builder class with the same fields, except mutable and nullable.<br /> The build method isn’t complex either. Just check every field was set and then return a new instance of our target class.<br /> Finally, the DSL entry method is also simple: we create a new builder, apply the initializer lambda and build.<br /> The full code for this is a bit long for this format, but you can find my implementation of AutoDSL <a href="https://github.com/F43nd1r/autodsl" target="_blank">here</a> .</p><h2>Problem: Default values</h2><p>Most people have two hands, so let’s set that as a default value:</p><div>

```kotlin
@AutoDSL
data class Person(val age: Int, val name: String, val hands: Int = 2)
```

</div><p>The builder we’d write by hand could then maybe look like this:</p><div>

```kotlin
data class PersonBuilder {
    var age: Int? = null
    var name: String? = null
    var hands: Int = 2
 
    fun build(): Person {
        check(age != null) { "age must be assigned." }
        check(name != null) { "name must be assigned." }
        return Person(age, name, hands)
    }
}
```

</div><p>However, there are two problems with this. First, we’re duplicating the default value. Especially if the default value is a more complex expression this might not be desirable. Second and more importantly, our annotation processing tools don’t give us access to default values at all, we can only see if there is one.</p><h3>How Kotlin default values work</h3><p>To find an alternative solution for this, we need to take a look at how default values are implemented in Kotlin internally. Here’s the decompiled Java equivalent of our Kotlin class (I’ve used IntelliJ here to view and decompile the Kotlin bytecode):</p><div>

```java
class Person {
 
  public Person(int age, @NotNull String name, int hands) {
    Intrinsics.checkNotNullParameter(name, "name");
    super();
    this.age = age;
    this.name = name;
    this.hands = hands;
  }
 
  // synthetic method
  public Person(int var1, String var2, int var3, int var4, DefaultConstructorMarker var5) {
    if ((var4 & 4) != 0) {
      var3 = 2;
    }
 
    this(var1, var2, var3);
  }
 
  //fields and methods excluded for brevity
}
```

</div><p>Okay, so at the top we see a normal constructor as we would expect, but at the bottom it gets interesting: here we have a <a href="https://www.baeldung.com/java-synthetic#synthetic-java" target="_blank">synthetic</a>  constructor with two additional arguments: an integer and a <code>DefaultConstructorMarker</code>. We’re going to ignore the second one, it’s just there to make sure the constructor signature can’t conflict with other existing constructors.<br /> The other (here <code>var4</code>) is interesting though – as we can see from the implementation it’s used as a bitfield to toggle the usage of our default values. In this case if the third bit is set (4 is <code>100</code> in binary), 2 is assigned to the third parameter, which is <code>hands</code>.</p><h3>Solution</h3><p>We can use this constructor with our own bitfield to set default values without ever needing to know what they are:</p><div>

```kotlin
class PersonBuilder {
    private var _defaultsBitField: Int = -1
 
    var age: Int? = null
 
    var name: String? = null
 
    var hands: Int by Delegates.observable(0) { _, _, _ -> _defaultsBitField = _defaultsBitField and -5 }
 
    fun build(): Person {
        check(age != null) { "age must be assigned." }
        check(name != null) { "name must be assigned." }
        val constructor = Person::class.java.getConstructor(Int::class.java, String::class.java, Int::class.java, Int::class.java, DefaultConstructorMarker::class.java)
        return constructor.newInstance(age, name, hands, _defaultsBitField, null)
    }
}
```

</div><p>Notable things are</p><ul><li><code>-1</code> is all ones in binary, meaning we turn on all default values initially. In this simple case we also could’ve just set it to 4</li><li><code>Delegates.observable</code> is just a nice <a href="https://kotlinlang.org/docs/delegated-properties.html#observable-properties" target="_blank">delegate</a>  to wrap a field to do something when it is written</li><li><code>-5</code> is the binary inverse of 4, which means in combination with <code>and</code> we can use it to turn off the third bit in our field, leaving all others untouched.</li><li>Since the constructor we want to call is synthetic, we can’t call it directly. Instead, we’re using <a href="https://www.oracle.com/technical-resources/articles/java/javareflection.html" target="_blank">reflection</a>  to find it and then call <code>newInstance</code>. The <code>DefaultConstructorMarker</code> can always be <code>null</code>.</li></ul><h2>Summary</h2><p>In this article we’ve talked about the advantages of Kotlin DSLs, namely inline logic and tooling support for syntax and type checking, formatting and autocompletion.<br /> We’ve also seen how to create a Kotlin DSL, and how to generate it using annotation processing.<br /> Lastly we looked at Kotlin internals to overcome one of the major pitfalls encountered when trying to generate a Kotlin DSL in this way.<br /> I’ve encountered more pitfalls on the way to building <a href="https://github.com/F43nd1r/autodsl" target="_blank">AutoDSL for Kotlin</a> , so let me know if you found this interesting, and I might write more on the topic.</p>

Generating Kotlin DSLs

<p>This article will give an introduction to Kotlin annotation processing using Kotlin Symbol Processing (KSP). We’ll take a look at KSP, compare it to the alternative KAPT, and implement a simple example.</p><p>I was inspired to write this post after multiple people asked me about KSP when reading my <a href="/wissens-hub/blog/generating-kotlin-dsls">last blog post</a> , which talked about generating Kotlin DSLs.</p><h2>What is annotation processing?</h2><p>Annotation processing in the JVM world usually refers to generating new source code at build time based on annotations present in the existing source code, which is also the scenario we’ll look at here.</p><p>A lot of variations are possible though: You could modify existing code instead of generating new code, as <a href="https://projectlombok.org/">Project Lombok</a>  does; generate non-code resources, like <a href="https://github.com/google/auto/tree/master/service">AutoService</a> ; or process annotations at runtime, akin to <a href="https://spring.io/projects/spring-boot">Spring Boot</a> .</p><h2>History of annotation processing</h2><p>JVM Annotations were introduced in Java 5 and with them the Annotation Processing Tool (apt) (not to be confused with the Advanced Packaging Tool (apt) commonly used on Linux systems).<br /> Apt was at the time a standalone program you had to run before invoking the Java Compiler (javac). That was of course not very convenient and apt was integrated into javac with Java 6.</p><p>After that, little changed for annotation processing in Java, but I’m not here to talk about Java anyway – this is about Kotlin. Annotation processing in Kotlin started with the release of Kotlin Annotation Processing Tool (kapt).</p><h2>What is kapt?</h2><p>Kapt is essentially a wrapper around apt: It generates equivalent Java stubs for all Kotlin classes and then calls javac to let it process the annotations within. It only needs to generate stubs (classes and methods without any instructions), because apt doesn’t look at instructions anyway.</p><p>This brought a major advantage: almost all annotation processors written for Java were suddenly compatible with Kotlin.</p><p>But it also has some major drawbacks:</p><ul><li>All Kotlin language features without direct equivalent in Java are effectively hidden from annotation processors. E.g. explicit nullability, primary constructors, properties, etc. (There are ways around this, parsing Kotlin metadata, but it’s clunky and too big of a topic to get into here.)</li><li>Stub generation is painfully slow, often taking longer than annotation processing itself</li></ul><p>For these reasons kapt is <a href="https://kotlinlang.org/docs/kapt.html" target="_blank">officially in maintenance mode</a>  and it is recommended to use KSP instead.</p><h2>What is KSP?</h2><p>KSP is a Kotlin compiler plugin aiming to provide a stable annotation processing API. This means while KSP needs to be updated for almost every Kotlin release, your annotation processor does not.</p><p>Aside from not supporting Java annotation processors, KSP has several advantages:</p><ul><li>Speed. According to <a href="https://kotlinlang.org/docs/ksp-overview.html" target="_blank">the documentation</a> , it can be up to 2 times faster than kapt.</li><li>Support for all Kotlin language features. For me the most important one here is explicit nullability.</li><li>Compatibility with non-JVM Kotlin targets, like Javascript or native. (Although improving the multiplatform experience is still on <a href="https://github.com/google/ksp#ongoing-and-future-works" target="_blank">the roadmap</a> .)</li></ul><p>Okay, so now you know KSP is awesome, but how do you use it?</p><h2>Getting started with KSP</h2><p>For this example I’m going to use a very simple gradle project (KSP supports maven as well if that’s more your jam).</p><div>

```kotlin
// settings.gradle.kts
include("app")
```

</div><div>

```kotlin
// build.gradle.kts
plugins {
    kotlin("jvm") version "1.6.21" apply false
}
 
subprojects {
    repositories {
        mavenCentral()
    }
}
```

</div><div>

```kotlin
// app/build.gradle.kts
plugins {
    kotlin("jvm")
    application
}
 
application {
    mainClass.set("MainKt")
}
```

</div><div>

```kotlin
// app/src/main/kotlin/main.kt
fun main() {
}
```

</div><div>

```kotlin
// app/src/main/kotlin/Printer.kt
interface Printer {
    fun `hello world!`()
}
```

</div><p>Our goal in this example will be to generate an implementation of <code>Printer</code>, which logs anything we write in the method name to the standard output.</p><h3>1. Setup modules</h3><p>Our annotation processor needs its own module, and we’ll create one for our custom annotation as well:</p><div>

```kotlin
// settings.gradle.kts
include("app")
include("annotation")
include("processor")
```

</div><div>

```kotlin
// annotation/build.gradle.kts
plugins {
    kotlin("jvm")
}
```

</div><p>Our processor will need access to the annotation and the KSP API. I’ll use <a href="https://square.github.io/kotlinpoet/" target="_blank">KotlinPoet</a>  to support code generation so add that as well:</p><div>

```kotlin
// processor/build.gradle.kts
plugins {
    kotlin("jvm")
}
 
dependencies {
    implementation("com.google.devtools.ksp:symbol-processing-api:1.6.21-1.0.5")
    implementation("com.squareup:kotlinpoet-ksp:1.11.0")
    implementation(project(":annotation"))
}
```

</div><p>Our app needs the annotation plus the processor. We’ll also make IntelliJ aware of our generated sources.</p><div>

```kotlin
// app/build.gradle.kts
plugins {
    kotlin("jvm")
    application
    id("com.google.devtools.ksp") version "1.6.21-1.0.5"
}
 
dependencies {
    implementation(project(":annotation"))
    ksp(project(":processor"))
}
 
application {
    mainClass.set("MainKt")
}
 
kotlin {
    sourceSets.main {
        kotlin.srcDir("build/generated/ksp/main/kotlin")
    }
    sourceSets.test {
        kotlin.srcDir("build/generated/ksp/test/kotlin")
    }
}
```

</div><p>If you’re only looking to use a processor instead of writing one, this last snippet is all you need, obviously replacing the dependencies with that processor.</p><h3>2. Add annotation</h3><p>To be able to process annotations, we need an annotation. Let’s call it <code>@GenerateConsolePrinter</code>:</p><div>

```kotlin
// annotation/src/main/kotlin/GenerateConsolePrinter.kt
@Retention(AnnotationRetention.SOURCE) // we only need this in source files
@Target(AnnotationTarget.CLASS) // should only go on classes or interfaces
annotation class GenerateConsolePrinter
```

</div><p>I’ll also annotate our <code>Printer</code> at this time, but you could do that later as well.</p><div>

```kotlin
// app/src/main/kotlin/Printer.kt
@GenerateConsolePrinter
interface Printer {
    fun `hello world!`()
}
```

</div><h3>3. Processor Setup</h3><p>We need to implement a <code>SymbolProcessor</code> (KSP support in KotlinPoet is in beta, so we need to opt in).</p><div>

```kotlin
// processor/src/main/kotlin/ConsolePrinterProcessor.kt
@OptIn(KotlinPoetKspPreview::class)
class ConsolePrinterProcessor(private val codeGenerator: CodeGenerator) : SymbolProcessor {
    override fun process(resolver: Resolver): List<KSAnnotated> {
        return emptyList()
    }
}
```

</div><p>And a <code>SymbolProcessorProvider</code>:</p><div>

```kotlin
// processor/src/main/kotlin/ConsolePrinterProcessorProvider.kt
class ConsolePrinterProcessorProvider : SymbolProcessorProvider {
    override fun create(environment: SymbolProcessorEnvironment): SymbolProcessor {
        return ConsolePrinterProcessor(environment.codeGenerator)
    }
}
```

</div><p>Which we then need to register using the fully qualified name for KSP to discover it.<br /> For this we add a file: <code>processor/src/resources/META-INF/services/com.google.devtools.ksp.processing.SymbolProcessorProvider</code></p>

```
ConsolePrinterProcessorProvider

```

<p>Now that we finished the setup, it’s time for implementation:</p><h3>4. Processor Implementation</h3><p>In this section I’m using KotlinPoet to describe the source code we want to generate. It provides convenient abstractions (<code>TypeSpec</code>, <code>FunSpec</code> and <code>FileSpec</code> below), which in my opinion make it the best tool for generating Kotlin sources from code, but you could use other methods (e.g. string templating) as well.</p><p>All the code in this section is part of the <code>process</code> function of <code>ConsolePrinterProcessor</code>.</p><p>First, we need to get all classes with our annotation:</p><div>

```kotlin
val annotatedClasses = resolver.getSymbolsWithAnnotation(GenerateConsolePrinter::class.java.name)
    .filterIsInstance<KSClassDeclaration>()
```

</div><p>We want to generate an implementation for each of these:</p><div>

```kotlin
for (annotatedClass in annotatedClasses) {
    val className = annotatedClass.simpleName.asString() + "Impl"
    val typeBuilder = TypeSpec.classBuilder(className)
        .addSuperinterface(annotatedClass.toClassName())
```

</div><p>Next, we want to override each method:</p><div>

```kotlin
    typeBuilder.addFunctions(annotatedClass.getDeclaredFunctions().map { function ->
        val name = function.simpleName.asString()
        FunSpec.builder(name)
            .addModifiers(KModifier.OVERRIDE)
            .addStatement("println(%S)", name)
            .build()
    }.toList())
```

</div><p>Our implementation here just prints out the method name with convenient <a href="https://square.github.io/kotlinpoet/#s-for-strings" target="_blank">KotlinPoet templating</a> .</p><p>And then we just need to write that all out into a file:</p><div>

```kotlin
    val fileSpec = FileSpec.builder(annotatedClass.packageName.asString(), className)
        .addType(typeBuilder.build())
        .build()
    codeGenerator.createNewFile(
            Dependencies(false, annotatedClass.containingFile!!),
            fileSpec.packageName,
            fileSpec.name
        )
        .writer()
        .use { fileSpec.writeTo(it) }
}
```

</div><p>The <code>Dependencies</code> here tell KSP which sources we used to generate that file, allowing it to only regenerate if relevant sources changed. This is great for performance on later builds.</p><p>And that’s it! Run a <code>gradle build</code> and have a look at our newly generated class:</p><div>

```kotlin
public class PrinterImpl : Printer {
  public override fun `hello world!`(): Unit {
    println("hello world!")
  }
}
```

</div><p>We can now use our generated class:</p><div>

```kotlin
// app/src/main/kotlin/main.kt
fun main() {
    PrinterImpl().`hello world!`()
}
```

</div><p>And test it with <code>gradle run</code>:</p><div>

```bash
[...]
> Task :app:run
hello world!
```

</div><p>Success! Of course in this example we needed to write more code than we saved, but I think it’s easy to imagine how you could use this to avoid boilerplate code. Annotation processing is best suited for situations where you need to write a lot of code with very similar structure, but not so similar that you could just extract it into a method.</p><p>You can find the complete sample code <a href="https://github.com/F43nd1r/ksp-example" target="_blank">here</a> .</p><h2>Summary</h2><p>In this blog post we’ve learned what kapt and KSP are, and the major advantages of KSP, namely speed, Kotlin language feature support and independence from the JVM. We’ve also seen how to use a KSP processor and how to create our own with KotlinPoet.</p><p>I hope the next time you repeat the same boilerplate code over and over again you’ll think of this post and maybe even write your own processor to save you all that effort.</p><p>If you’re currently using KAPT, you can also check if your processors already support KSP for a nice build speed boost.</p>

An Introduction to Kotlin Symbol Processing

IT Developer und Consultant

IT Developer and Consultant

<p>In diesem Artikel erkläre ich, was ein Kubernetes Operator ist und wie er aufgebaut ist. Anschließend zeige ich euch, wie man seinen ersten eigenen Kubernetes Operator in Go schreibt.</p><h2><b>Was ist ein Kubernetes Operator</b></h2><p>Ein Kubernetes Operator hilft, eine Applikation, die auf Kubernetes läuft zu verpacken, zu verwalten und zu deployen. Der Operator wird über das Standard-Tooling (kubectl und Kubernetes API) deployed und verwaltet.</p><p>Die Idee ist, das Wissen von Operations-Teams in Software zu verpacken, die benötigte Schritte automatisiert ausführen kann.</p><p>So kann zum Beispiel ein Elasticsearch-Cluster erstellt werden, bei dem man die Größe des Clusters automatisch anpassen kann. Auch können automatisiert Updates installiert werden. Das geschieht dabei über wenige Angaben, wie die zu verwendete Version oder die Anzahl Nodes. Der Operator übernimmt dabei den Rest. So ist es möglich, mit relativ wenig Angaben und ohne Administrations-Wissen das Cluster aufzusetzen.</p><p>Operatoren befolgen das Prinzip des “Maturity Model”. Dies bedeutet, der Operator übernimmt am Anfang nur einen kleinen Teil der Arbeit. Je ausgereifter der Operator, desto mehr Funktionen (wie automatisierte Upgrades und Backups, auf Fehler reagieren) übernimmt dieser. Die nachfolgende Grafik veranschaulicht das gut.</p><p>Operatoren können mithilfe des <a href="https://github.com/operator-framework/operator-sdk" target="_blank">Operator-SDK</a>  auf Basis von Helm Charts, Ansible Playbooks oder Go-Code erstellt werden. In diesem Blogpost erstellen wir einen Operator mit Go. Falls ihr lieber wissen wollt wie man einen Operator mithilfe eines Helm Charts erstellt, empfehle ich euch den <a href="/wissens-hub/blog/kubernetes-operators-helm" target="_blank">Blogpost “Deploying Helm Charts without Tiller”</a>  meines Kollegen Roman.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/rBNrV6kuT4615SJVHhmA" target="_blank"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/rBNrV6kuT4615SJVHhmA" alt="Maturity Model" width="700" height="344" /></a> <p>Visualisierung “Maturity Model” Quelle: https://github.com/operator-framework/operator-sdk</p></div><h2><b>Was sind CR/CRDs?</b></h2><p>Ein Operator besteht aus einer oder mehreren <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/" target="_blank">Custom Resources</a>  (CR) und einer eigenen Logik in Form eines Controllers. Der <a href="https://kubernetes.io/docs/concepts/architecture/controller/" target="_blank">Controller</a>  einer Ressource ist dafür verantwortlich, den aktuellen Zustand des Kubernetes-Cluster näher zum gewünschten Zustand zu bringen.</p><p>CR sind dabei eine Möglichkeit, Kubernetes um eigene Ressourcen zu erweitern. Diese können dann wie normale Kubernetes-Objekte verwendet werden. Man erweitert so die API von Kubernetes um eigene Objekte. Diese lassen sich wie andere Objekte mit Standard-Tools wie kubectl konfigurieren.</p><p>Die Spezifikation einer Ressource befindet sich in der Custom Resource Definition (CRD).</p><p>Das Operator-SDK unterstützt bei der Erstellung einer CRD, indem es anpassbare, bereits funktionale Code-Templates generiert. Die Business-Logik für diese neue CRD kann direkt im Operator in Form von Go-Code umgesetzt werden.</p><p>CRD sind, wie es der Name bereits verrät, eine Möglichkeit, Kubernetes um eigene Ressourcen zu erweitern. Aufgrund dieser Definition können Entwickler eigene CR in Form von YAML-Dateien benutzen/schreiben und diese wie andere Kubernetes-API-Objekte über kubectl hinzufügen. Der Operator übernimmt dann den Rest und macht die nötigen Änderungen im Cluster selbstständig.</p><p> </p><h3>Primäre/Sekundäre Ressourcen</h3><p>Bevor wir jetzt unseren eigenen Operator entwickeln, gibt es noch eine Sache zu klären. Der Operator kann verschiedene Ressourcen überwachen. Üblicherweise wird zwischen primären und sekundären Ressourcen unterschieden. Als primäre Ressourcen zählen die selbst erstellten CR. Sekundäre Ressourcen sind Ressourcen, die üblicherweise durch den Operator erstellt und verwaltet werden, um den in den CR beschriebenen Zustand zu erreichen.</p><p>Als Beispiel nehmen wir die Kubernetes-Komponente “Deployment”. In einem Deployment kann man eine Anzahl an Replikas definieren. Die Zahl gibt dabei vor, wie viele Pods erstellt werden sollen. In diesem Beispiel wäre das Deployment die primäre und die Pods die sekundäre Ressource.</p><p>Die primären Ressourcen (Deployments) entsprechen in diesem Beispiel der CR und die sekundären Ressourcen sind die zu überwachenden Objekte (Anzahl Pods), die zur Erfüllung der Funktion des Operators dienen.</p><p> </p><h2>Eigener Operator mit Operator-SDK</h2><p><i>Die folgende Anleitung basiert auf der aktuell neuesten Version v0.11.0 des </i><a href="https://github.com/operator-framework/operator-sdk" target="_blank"><i>Operator-SDK</i></a> <i>. Das SDK ist für Linux und Mac verfügbar. Eine Windows-Version ist aktuell </i><a href="https://github.com/operator-framework/operator-sdk/blob/v0.10.0/doc/dev/release.md#operating-systems-and-architectures" target="_blank"><i>nicht geplant.</i></a> </p><h3>Vorbereitungen</h3><p>Ihr solltet das Operator-SDK bereits installiert haben. Eine Anleitung und die dazugehörigen <a href="https://github.com/operator-framework/operator-sdk#prerequisites" target="_blank">Voraussetzungen</a>  gibt es im offiziellen Github-Repo. Überprüfen ob ihr es erfolgreich installiert habt, könnt ihr mit folgendem Befehl:</p>

```
operator-sdk version
```

<p> </p><h3>Projekt-Erstellung</h3><p>Wir erstellen nun einen PodReplica Operator. Dieser soll wie ein Deployment für ein bestimmtes Image (Busybox, sleep 1h) fungieren. Man soll über das Feld “size” steuern können, wie viele Pods mit dem Busybox-Image vorhanden sein sollten. Wir verwenden hier mit Absicht ein einfaches Beispiel, da es darum geht zu lernen, wie ein Operator funktioniert.</p><p>Folgende Befehle werden im Terminal ausgeführt um ein Projekt anzulegen (den example-user sollte man dabei durch den eigenen Github-Usernamen ersetzen):</p>

```
export GO111MODULE=on # Falls eigene Go-Version < 1.13
operator-sdk new podreplica-operator --repo github.com/example-user/podreplica-operator
cd podreplica-operator
```

<p>Das Operator-SDK generiert nun folgende Projektstruktur:</p>

```
├── build
│   ├── bin
│   │ ├── entrypoint
│   │ └── user_setup
│   └── Dockerfile
├── cmd
│   └── manager
│       └── main.go
├── deploy
│   ├── operator.yaml
│   ├── role_binding.yaml
│   ├── role.yaml
│   └── service_account.yaml
├── go.mod
├── go.sum
├── pkg
│   ├── apis
│   │ └── apis.go
│   └── controller
│       └── controller.go
├── tools.go
└── version
└── version.go
```

<p>Diese beinhaltet zum Beispiel ein Dockerfile (im Ordner build), die später zu deployenden YAML-Dateien (im Ordner deploy) und einige Go-Files.</p><h3>CRD erstellen</h3><p>Als nächstes fügen wir unsere neue CRD hinzu:</p>

```
operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=PodReplica
```

<p>Dieser Befehl registriert die neue CR “PodReplica” und deren APIs. In der generierten CRD (deploy/crds/app.example.com_podreplicas_crd.yaml) sieht man auch, dass verschiedene Namen für die neue Ressource reserviert werden:</p><p></p><p>Wir als Entwickler geben den Zielzustand (Spec) eines Objekts in Kubernetes vor, während der Status hingegen den aktuellen Zustand des Objekts zeigt. Der Kubernetes-Scheduler versucht dabei den Zielzustand zu erreichen und zu halten. Dies geschieht mithilfe des Reconcilation Loop. Dieser ist ein zentraler Bestandteil eines Operators – dazu aber später mehr.</p><p>Für unsere eigene CR müssen wir die dazugehörigen Felder definieren. Diese werden in den jeweiligen Go-Structs angegeben (zu finden unter pkg/apis/app/v1alpha1/podreplica_types.go).</p><p></p><p>Ein Feld besteht aus einem Namen, dem Typ und dem Namen, den es bei der De-/Serialisierung zu JSON haben sollte.</p><p>Nach jeder Anpassung dieser *types.go-Dateien müssen die folgenden zwei Befehle ausgeführt werden:</p>

```
// Updated den generierten Code
operator-sdk generate k8s
```



```
// Updated die CRD und die OpenAPI Validierungen
operator-sdk generate openapi
```

<p>Nach der Ausführung dieser beiden Befehle sieht man, dass unsere CRD (deploy/crds/app.example.com_podreplicas_crd.yaml) um die entsprechenden Felder erweitert wurde.</p><h3>Controller erstellen</h3><p>Nachdem die CRD erstellt wurde, wollen wir nun auch noch den dazugehörigen Controller erstellen, der unsere Businesslogik enthält:</p>

```
operator-sdk add controller --api-version=app.example.com/v1alpha1 --kind=PodReplica
```

<p>In diesem generierten Controller (pkg/controller/podreplica/podreplica_controller.go) müssen wir angeben, welche Ressourcen (siehe Primäre/Sekundäre Ressourcen) wir überwachen wollen. Für diese Ressourcen wird der Controller bei jeder Änderung benachrichtigt. Standardmäßig werden die erstellten CR (in unserem Fall die PodReplicas) und die daraus resultierenden Pods überwacht (siehe Code-Ausschnitt).</p>

```
// Watch for changes to primary resource PodReplica
err = c.Watch(&source.Kind{Type: &appv1alpha1.PodReplica{}}, &handler.EnqueueRequestForObject{})

// Watch for changes to secondary resource Pods and requeue the owner PodReplica
err = c.Watch(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForOwner{
IsController: true,
OwnerType:    &appv1alpha1.PodReplica{},
})
```

<p><i>Ausschnitt aus Funktion add</i></p><p>Für unseren Fall passt dies bereits und wir müssen hier nichts weiter anpassen.</p><p>Nun fehlt nur noch die eigentliche Logik, um Pods hoch- und runterzufahren und die Namen der Pods im Status unserer CR zu pflegen. Das geschieht in der Funktion “reconcile”. Diese Methode wird immer aufgerufen, wenn sich eine der überwachten Ressourcen ändert.</p><p>Wir schauen uns exemplarisch den Code zum Skalieren von Replikas an.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/gihmaI4SSMACxpSdSvgF" target="_blank"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/gihmaI4SSMACxpSdSvgF" alt="pod replica code sample" width="1245" height="513" /></a> <p>Code für Skalierung der Pods</p></div><p><span>(1)</span> Sobald die Anzahl der aktuell laufenden Pods nicht der definierten Größe aus der Spec entspricht, wird ein Pod-Objekt erstellt. <span><br /> (2)</span> Bewirkt, dass unser Operator die Kontrolle über die Laufzeit des Objektes hat und bei Änderungen über diesen Pod benachrichtigt wird (sofern wie bei uns auf diese Ressource eine Watch mit EnqueueRequestForOwner vorhanden ist).<br /> <span>(3)</span> Als letztes wird über den Client der Pod im Kubernetes-Cluster angelegt.</p><p>Ich gehe nun nicht weiter auf die Details der Implementierung ein (der Code ist auf <a href="https://github.com/WtfJoke/podreplica-operator" target="_blank">Github</a>  verfügbar).</p><p>Zusammengefasst ist wichtig, sich über folgende Punkte klar zu sein:</p><ol><li>Die Funktion “Reconcile “ wird bei jeder Änderung der überwachten Ressourcen aufgerufen.</li><li>Erstellt man neue Objekte, sollte die eigene Ressource (dh. PodReplica) der Besitzer der Ressource sein. Dies geschieht mit der Funktion controllerutil.SetControllerReference.<br /> Damit werden beim Löschen der PodReplica auch alle dazugehörigen Pods automatisch gelöscht.</li><li>Die Reconcile-Funktion sollte idempotent sein. Zu einem gegebenen Zustand sollte es keinen Unterschied machen, wie oft man die Funktion aufruft. Sie sollte immer dasselbe Resultat liefern.<br /> Man sollte zum Beispiel nicht zwischen verschiedenen Events (Create/Delete/Update) unterscheiden.</li><li>Um mit der Kubernetes-API zu interagieren, kann man den Client verwenden, welcher vom Operator-SDK zur Verfügung gestellt wird.</li></ol><h2>Build, Run und Publish</h2><p>Es gibt zwei Möglichkeiten, wie man den Operator nun starten kann:</p><h3>1. Operator lokal starten</h3><p>Während des Entwicklungszyklus macht es mehr Sinn, den Operator lokal zu testen. Ich habe dafür <a href="https://kubernetes.io/docs/setup/learning-environment/minikube/" target="_blank">minikube</a>  verwendet, werde aber auf die Installation hier nicht weiter eingehen.</p><p>Als erstes wird die neue CRD in Kubernetes registriert:</p>

```
kubectl apply -f deploy/crds/app.example.com_podreplicas_crd.yaml
```

<p>Nachfolgend setzt man noch den Namen des Operators in einer Umgebungsvariablen:</p>

```
export OPERATOR_NAME=podreplica-operator
```

<p>Anschließend startet man den Operator mit folgendem Befehl:</p>

```
operator-sdk up local --namespace=default
```

<p>Oder mit aktiviertem Debugging (<a href="https://github.com/go-delve/delve" target="_blank">Delve</a>  sollte installiert sein):</p>

```
operator-sdk up local --namespace=default --enable-delve
```

<p> </p><h3>2. Als Deployment in einem Cluster</h3><p>Um den Operator in einem Cluster zu verwenden, müssen wir diesen zuerst bauen. Danach pushen wir das resultierende Docker Image in eine beliebige Docker Registry. Wir benutzen <a href="https://hub.docker.com" target="_blank">https://hub.docker.com</a>  als Registry.</p>

```
operator-sdk build <user>/podreplica-operator:v0.0.1
# Linux
sed -i 's|REPLACE_IMAGE|<user>/podreplica-operator:v0.0.1|g' deploy/operator.yaml
# OSX:
sed -i "" 's|REPLACE_IMAGE|<user>/podreplica-operator:v0.0.1|g' deploy/operator.yaml

docker push <user>/podreplica-operator:v0.0.1
```

<p>Die Befehle ersetzen den Platzhalter “REPLACE_IMAGE” mit dem gebauten Docker Image <image:tag>. In der Datei deploy/operator.yml sollte nun auf das eben gebaute Image verwiesen werden.</p><p>Im Cluster muss die CRD als erstes registriert werden:</p>

```
kubectl apply -f deploy/crds/app.example.com_podreplicas_crd.yaml
```

<p>Danach erstellt man die nötigen Berechtigungen (RBAC) und den Operator selbst:</p>

```
kubectl create -f deploy/service_account.yaml
kubectl create -f deploy/role.yaml
kubectl create -f deploy/role_binding.yaml
kubectl create -f deploy/operator.yaml

```

<p>Überprüfen, ob das Operator-Deployment läuft:</p>

```
kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE AGE
podreplica-operator   1/1     1            1         58s
```

<p>Überprüfen, ob der Operator-Pod läuft:</p>

```
kubectl get pods
NAME                                   READY   STATUS    RESTARTS AGE
podreplica-operator-5cd9b67b79-lg7mc   1/1     Running   0        59s
```

<p> </p><h3>PodReplica erstellen</h3><p>Jetzt steht nichts mehr im Wege, eigene PodReplicas zu erstellen. Wir haben eine Beispiel-Definition einer PodReplica im Ordner (deploy/crds) und können sie mit nachfolgendem Befehl erstellen:</p>

```
kubectl apply -f deploy/crds/app.example.com_v1alpha1_podreplica_cr.yaml
```

<p>Jetzt stellen wir noch sicher, dass die Ressource existiert und die Namen der Replica(Pods) eingetragen sind:</p>

```
kubectl get podreplica/example-podreplica -o yaml
```

<p></p><p>Überprüfen, ob alle Pods funktionieren:</p>

```
kubectl get pods
NAME                                  READY   STATUS    RESTARTS AGE
example-podreplica-pod2hm4w           1/1     Running   0        7m17s
example-podreplica-pod5g7kh           1/1     Running   0        7m17s
example-podreplica-podl55nq           1/1     Running   0        7m17s
podreplica-operator-5cd9b67b79-lg7mc  1/1     Running   0        37m
```

<p> </p><h4>Größe anpassen</h4><p>Wir können die “Size” in der .yml-Datei deploy/crds/app_v1alpha1_podreplica_cr.yml anpassen und die Änderung hinzufügen:</p>

```
cat deploy/crds/app.example.com_v1alpha1_podreplica_cr.yaml

apiVersion: app.example.com/v1alpha1
kind: PodReplica
metadata:
  name: example-podreplica
spec:
  size: 4
```



```
kubectl apply -f deploy/crds/app.example.com_v1alpha1_podreplica_cr.yaml
kubectl get pods

NAME                                   READY   STATUS    RESTARTS AGE
example-podreplica-pod5g7kh            1/1     Running   0        28m
example-podreplica-podjhdhl            1/1     Running   0        3m31s
example-podreplica-podl55nq            1/1     Running   0        28m
example-podreplica-podws7cp            1/1     Running   0        3m31s
podreplica-operator-5cd9b67b79-lg7mc   1/1     Running   0        58m

```

<p> </p><h4>PodReplica/Operator entfernen</h4><p>So leicht wie das Hinzufügen der Ressource war, ist auch das Löschen:</p>

```
kubectl delete -f deploy/crds/app.example.com_v1alpha1_podreplica_cr.yaml
```

<p>Alle relevanten Pods werden nun automatisch heruntergefahren/gelöscht.<br /> Den Operator und seine Berechtigungen löscht man mit folgenden Befehlen:</p>

```

kubectl delete -f deploy/crds/app.example.com_podreplicas_crd.yaml
kubectl delete -f deploy/operator.yaml
kubectl delete -f deploy/role_binding.yaml
kubectl delete -f deploy/role.yaml
kubectl delete -f deploy/service_account.yaml

```

<p> </p><h2>Fazit</h2><p>Wir haben es geschafft! Wir haben unseren ersten eigenen Kubernetes Operator erstellt und deployed 😀</p><p>Den kompletten Code findet ihr auf <a href="https://github.com/WtfJoke/podreplica-operator" target="_blank">Github</a> .</p><p>Falls ihr noch weitere Inspiration braucht, gibt es noch einen zweiten Beispiel-Operator der “Chaos” stiftet, indem dieser Pods mit einem vordefinierten Prefix löscht. Diesen gibt es ebenfalls auf <a href="https://github.com/WtfJoke/ordered-chaos-monkey-operator" target="_blank">Github (ChaosPod Operator)</a> .</p><p>Falls ihr weitere Infos zum Operator-SDK benötigt, ist das <a href="https://github.com/operator-framework" target="_blank">offizielle Repository</a>  ein guter Start. Weitere, auch produktiv genutzte Operators können auf folgenden Seiten gefunden werden:</p><p><a href="https://operatorhub.io/" target="_blank">https://operatorhub.io/</a> <br /> <a href="https://github.com/operator-framework/awesome-operators" target="_blank">https://github.com/operator-framework/awesome-operators</a> </p>

Kubernetes Operator: Operations-Wissen als Code

60% CO2-Emissionen sparen – Unsere Reise zur nachhaltigen codecentric-Website

<p>Imagine the following scenario: You set up your GitHub Actions in your repository. And it’s all cool until you want to access your cloud provider resources. Now you might be tempted to create an access key and secret access key, place it as a secret in your repository and forget about it. However, this can have drawbacks, for example your credentials could get leaked without your knowledge and your AWS account could be accessed by third parties. Additionally you probably haven’t thought about how to rotate these credentials. You might ask yourself: Isn’t there a better way to access your cloud resources without storing these credentials anywhere…?</p><p>Fear no more, GitHub and your favorite cloud provider got you covered!</p><p>When you wanted to access cloud resources (AWS, Azure, Google Cloud Platform/GCP) from your GitHub Actions, up until recently, you needed to store some sort of credentials (such as an access key and secret access key) as secrets in your repository. Not anymore!</p><p>GitHub <a href="https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/" target="_blank">announced</a>  the possibility to get rid of this type of credentials.<br /> Instead, you can configure your GitHub Actions so that an <a href="https://connect2id.com/learn/openid-connect" target="_blank">OpenID Connect</a>  <a href="https://jwt.io/introduction" target="_blank">JSON Web Token</a>  (or short OIDC JWT) is generated by GitHub while your workflow runs.</p><p>In your cloud provider, you can register GitHub as an Identity Provider. This way, the JWT generated by GitHub is allowed to access your cloud account.</p><p>GitHub provides <a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments" target="_blank">instructions</a>  on how to use this feature for AWS, Azure and GCP. In this blog post I will focus on AWS, as it’s the most widely used cloud provider and the one I’m most familiar with.</p><h2><strong>How the authentication between GitHub and AWS works</strong></h2><p>So before we get our hands dirty, let’s have a look at how the secretless connection using OIDC works:</p><p><img width="921" height="892" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/RpkXpVLaRjWlHWaFKvjg" alt="diagram how github connects to aws using oidc" /></p><ol><li>In AWS (or your cloud provider of choice), create a trust relationship between your cloud role and the tokens issued by GitHub and used in your GitHub Actions workflow.</li><li>A GitHub Action (such as <a href="https://github.com/aws-actions/configure-aws-credentials" target="_blank">aws-actions/configure-aws-credentials</a> ) can request a signed JWT with multiple claims during an action job/workflow run.</li><li>GitHub OIDC provider issues a signed JWT with multiple claims to the github actions workflow.</li><li>The action sends the JWT and the requested role to AWS.</li><li>The JWT is validated in AWS and AWS sends a short-lived access token in exchange. With this token, it’s possible to access AWS resources.</li></ol><p>The JWT contains claims e.g. the <a href="https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims" target="_blank">standard claims</a>  like “subject”. Each claim holds different information about the token itself. GitHub adds additional custom claims like ”reference”. You can find an example of the available claims <a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#understanding-the-oidc-token" target="_blank">here</a> . These claims will later be used for the trust conditions.</p><h2><strong>Creating the trust</strong></h2><p>Now that we know how it works theoretically, we can start coding / setting it up in code.<br /> The following infrastructure code examples are available in <a href="https://github.com/WtfJoke/aws-gh-oidc">a demo repository</a> . The examples are written in both <a href="https://aws.amazon.com/de/cdk/">AWS CDK</a>  and <a href="https://www.terraform.io/">Terraform</a> . You can also click through the AWS management console as well, but please don’t hold me accountable for your next AWS bill ;).</p><p>In order to access AWS resources from GitHub, we need to make sure we have</p><ul><li>registered GitHub as a (trusted) OIDC Provider in AWS by<ul><li>including under which conditions a GitHub token is considered trustworthy (trust condition),</li></ul></li><li>created the resource itself,</li><li>created an AWS IAM role which can be assumed by your CI pipeline,<ul><li>including a policy for that role which defines the AWS resources which are allowed to access.</li></ul></li></ul><h3>Create OIDC Provider connection</h3><p>First, lets create the OIDC Provider in AWS:</p><ul><li>In IAM → Identity providers → Add provider</li><li>Provider URL: https://token.actions.githubusercontent.com<br /> Audience: sts.amazonaws.com</li></ul><p>In CDK:</p><div>

```javascript
const githubOIDCProvider = new iam.OpenIdConnectProvider(
  this,
  "GithubActions",
  {
	url: "https://token.actions.githubusercontent.com",
	clientIds: ["sts.amazonaws.com"],
  }
);
```

</div><p>This is how you tell AWS to allow tokens issued by GitHub to be accepted by AWS.<br /> The audience/clientId “sts.amazonaws.com” is used by the official AWS GitHub Action <a href="https://github.com/aws-actions/configure-aws-credentials" target="_blank">aws-actions/configure-aws-credentials</a> .</p><h3>Create a resource</h3><p>The next step is to create a resource in AWS for demonstrating the access during a workflow run.<br /> I’ve chosen the service <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html" target="_blank">Parameter Store</a>  given that it’s cheap (most of the time free) and simple to store and retrieve a string. Give it a name (“hello_aws-gh-oidc” in this case) and a value and you’re good to go.</p><div>

```javascript
// Something to read for the pipeline :)
const helloParameter = new ssm.StringParameter(this, "HelloParameter", {
  description: `Sample value for demo purpose of project ${projectName}`,
  parameterName: `hello_${projectName}`,
  stringValue: "Hi from aws :wave:",
});
```

</div><h3>Create an IAM role</h3><p>The last thing we need to do in our AWS account is to create an IAM role which is allowed to access the parameter you just created (permission policy) and is allowed to be assumed by GitHub (trust policy).</p><p>When using the GitHub OIDC approach, the trust policy is used to verify the content/claims of the issued JWT from GitHub.<br /> The most common way is to verify the content of the subject and audience claim.</p><ol><li>The subject claim provides GitHub Actions job metadata, e.g. the branch name in case of a git push event. It looks different depending on which event triggered your job (GitHub provides example subjects <a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#example-subject-claims" target="_blank">here</a> ).</li><li>The audience claim provides the URL of the organization or repository owner by default. The <a href="https://github.com/aws-actions/configure-aws-credentials" target="_blank">aws-actions/configure-aws-credentials</a>  action overrides the audience claim with the value “sts.amazonaws.com”</li></ol><p>In CDK you define a secure IAM policy like this:</p><div>

```javascript
const githubActionsRole = new iam.Role(this, "GithubActionsRole", {
  // Trust policy
  assumedBy: new iam.WebIdentityPrincipal(
	githubOIDCProvider.openIdConnectProviderArn,
	{
	  StringEquals: {
		// Only allow tokens issued by aws-actions/configure-aws-credentials
		"token.actions.githubusercontent.com:aud": audience,
		// Only allow specified branches to assume this role
		"token.actions.githubusercontent.com:sub":
		  allowedBranchPatternToPush,
	  },
	}
  ),
  roleName: "aws-gh-oidc", // same as in .github/workflows/hello.yml
  description: `Role to assume from github actions pipeline of ${projectName}`,
});
 
// Permission policy
helloParameter.grantRead(githubActionsRole);
```

</div><p>This results in the following secure IAM role trust policy:</p><div>

```javascript
{
    "Statement": [
        {
            // ...
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:sub": [
                        "repo:WtfJoke/aws-gh-oidc:ref:refs/heads/main",
                    ],
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
```

</div><p><em>Only a push to the main branch of the repository named aws-gh-oidc from user WtfJoke can assume the role.</em></p><p>Please make sure you define your IAM role trust policies carefully.<br /> Above is an example for a secure policy. Below is an example of an unsecure policy (the policy allows assuming the IAM role from other users repositories):</p><p><strong>Unsecure</strong> IAM role trust policy:</p><div>

```javascript
{
    "Statement": [
        {
            // ...
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:sub": [
                        "repo:*/aws-gh-oidc:ref:refs/heads/main", // UNSECURE DO NOT USE
                    ],
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
```

</div><p><em>A push to the main branch of a repository named aws-gh-oidc from <strong>any</strong> users can assume the role</em></p><h2><strong>Setting up the Pipeline/Workflow</strong></h2><p>Now that we have everything set up on AWS, the only thing left to do is to create a simple GitHub workflow which assumes the AWS role, and reads the parameter store parameter and prints the value. The final workflow looks like this:</p><div>

```yaml
name: Hello from AWS

on:
  push:
  
permissions:
  id-token: write

jobs:
  greeting:
    runs-on: ubuntu-latest

    steps:
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-region: eu-central-1
        role-to-assume: arn:aws:iam::589918074230:role/aws-gh-oidc

    - name: Print AWS SSM Parameter
      run: aws ssm get-parameter --name=hello_aws-gh-oidc --query Parameter.Value

    - name: Print assumed role
      run: aws sts get-caller-identity
```

</div><p>We will go through all important steps in the workflow below.</p><h3>Adding permissions</h3><p>We need to allow the workflow to request a signed JWT from GitHub by setting the permissions:</p><div>

```yaml
permissions:
  id-token: write
```

</div><p><em>Note: Depending on your use case, you might need additional permissions for your workflow. (see GitHub Docs for the <a href="https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token" target="_blank">complete list of default permissions</a> ).</em></p><h3>Logging in at AWS</h3><p>To exchange a GitHub JWT for a short-lived AWS access token, we can use the <a href="https://github.com/aws-actions/configure-aws-credentials" target="_blank">aws-actions/configure-aws-credentials</a>  action.<br /> We need to pass the role (ARN) which we want to assume and have created earlier.</p><div>

```yaml
- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
	aws-region: eu-central-1
	role-to-assume: arn:aws:iam::589918074230:role/aws-gh-oidc
```

</div><h3>Interacting with AWS resources</h3><p>Finally we can interact with AWS by using the AWS CLI as usual.<br /> In our case, we can read the parameter we created earlier and print it:</p><div>

```yaml
- name: Print AWS SSM Parameter
  run: aws ssm get-parameter --name=hello_aws-gh-oidc --query Parameter.Value
```

</div><h3>Running the secretless workflow</h3><p>Now that we have set everything up and as soon as we push any commit to the repository, we should see a successful action run like the following:<br /> <img loading="lazy" width="752" height="265" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/i4P7FGdYTSWTHdzAPcX1" alt="github action prints hello world from aws parameter" /></p><h2><strong>What’s next?</strong></h2><p>By now, you have learned everything to get started using OIDC to access your cloud resources.<br /> The most important takeaways:</p><ol><li>Favor using OIDC over regular credentials to access cloud resources.</li><li>Double check the trust policies of your roles.</li></ol><p>You can find all the code examples in <a href="https://github.com/WtfJoke/aws-gh-oidc" target="_blank">this demo repository</a> .</p><p>Although our use case was simple by just printing a single string parameter, only your imagination limits the things you can do in your workflow. 🌈😄</p><p>There are even more advanced use cases: As soon as a pull request (PR) is opened, your application is deployed in a temporary environment (just like <a href="https://vercel.com/docs/concepts/git/vercel-for-github" target="_blank">vercel</a>  does) with the changes for the PR, so people can test the changes before it gets deployed to a live environment.</p><p>In contrast a push to the main branch is only allowed to deploy/update the existing live environment.<br /> For different use cases, you can use different IAM roles and limit their access to the required minimum.</p><p>If you have comments or questions feel free to reach out.</p><p><em>Sources</em>:<br /> <a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect" target="_blank">https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect</a> <br /> <a href="https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token" target="_blank">https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token</a> <br /> <a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services" target="_blank">https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services</a> <br /> <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html" target="_blank">https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html</a> <br /> <a href="https://github.com/aws-actions/configure-aws-credentials#assuming-a-role" target="_blank">https://github.com/aws-actions/configure-aws-credentials#assuming-a-role</a> <br /> <a href="https://connect2id.com/learn/openid-connect" target="_blank">https://connect2id.com/learn/openid-connect</a> <br /> <a href="https://jwt.io/introduction" target="_blank">https://jwt.io/introduction</a> </p>

Secretless connections from GitHub Actions to AWS using OIDC

IT Consultant / Software-Engineer / Architect 

Grafik Designerin

<p>Wer sich dieser Tage mit dem Themengebiet Big- oder Fast-Data beschäftigt, kommt am sogenannten <a href="https://www.codecentric.de/leistungen/loesungen/technologie-treibt-veraenderung/big-data/smack/">SMACK-Stack</a>  nicht vorbei. Dieser Artikel demonstriert, wie einfach es ist, den kompletten Stack in der AWS aufzubauen. Im ersten Schritt setzen wir ein DC/OS-Mesos-Cluster auf. Dieses Cluster dient uns als Grundinfrastruktur, in die wir Kafka, Cassandra und Spark installieren. Zum Abschluss starten wir einfache Ingestion- und Digestion-Jobs, die Live-Daten durch den Stack pumpen. SMACK ist zwar ein griffiges Akronym, aus logischen Gründen wird die Reihenfolge allerdings MCKSA(S) lauten.</p><p><span></span></p><h2>Schritt 0: Vorbedingungen</h2><p>Um die folgenden Beispiele selbst ausführen zu können, benötigt Ihr folgendes:</p><ul><li>Vagrant und Virtualbox</li><li>Ansible</li><li>Einen AWS-Account. Seid gewarnt: Es fallen Nutzungsgebühren an</li><li>Ein EC2-Schlüsselpaar dcos-intro</li><li>Einen Twitter-Zugangsschlüssel</li></ul><p>Der Quellcode ist auf <a href="https://github.com/ftrossbach/intro-to-dcos">https://github.com/ftrossbach/intro-to-dcos</a>  verfügbar.</p><h2>Schritt 1: <small>s</small><strong>M</strong><small>ack</small> steht für Mesos</h2><p>DC/OS ist ein verteiltes Betriebssystem für Rechenzentren und basiert auf dem Apache-Projekt Mesos. Seit Version 1.7 ist DC/OS <a href="https://dcos.io/" target="_blank">Open Source</a> . Unter <a href="https://downloads.dcos.io/dcos/EarlyAccess/aws.html" target="_blank">https://downloads.dcos.io/dcos/EarlyAccess/aws.html</a>  stellt das Projekt Vorlagen für AWS CloudFormation bereit, mit denen man einfach ein DC/OS-Cluster erstellen kann. Das folgende Diagramm gibt einen Überblick über die AWS-Ressourcen, die für dieses Beispiel erstellt werden:</p><p> </p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4qe10tJYSKNCwxRDDZ6s"><img width="700" height="570" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4qe10tJYSKNCwxRDDZ6s" alt="asdf" /></a> <p>Die CloudFormation-Vorlage erzeugt diese AWS-Ressourcen.</p></div><p>Das öffentliche Subnetz ist vom Internet aus erreichbar. Es enthält einen Mesos-Master sowie einen öffentlichen Mesos-Agenten. Für dieses Beispiel reicht ein Master aus, da wir keine Hochverfügbarkeit des Clusters benötigen. DC/OS bietet aber auch Vorlagen für Hochverfügbarkeitsszenarien. Das private Subnetz ist nur aus dem öffentlichen Subnetz erreichbar. Es enthält 5 Mesos-Agenten, auf die sämtliche Applikationen ausgerollt werden können, die nicht über das Internet verfügbar sein müssen. CloudFormation-Vorlagen werden schnell unübersichtlich. Der <a href="/wissens-hub/blog/aufbau-eines-mesosphere-dcos-data-center-operation-system-clusters-mit-terraform">Blogartikel meines Kollegen Bernd Zuther</a>  beschreibt, wie man statt CloudFormation Terraform benutzen kann.</p><p>Um den Stack aufzubauen, müsst Ihr zunächst das Beispielprojekt auf <a href="https://github.com/ftrossbach/intro-to-dcos" target="_blank">Github</a>  klonen. Zusätzlich ist es notwendig, dass Ihr folgende Umgebungsvariablen setzt:</p><ul><li>DCOS_INTRO_AWS_REGION</li><li>DCOS_INTRO_AWS_ACCESS_KEY</li><li>DCOS_INTRO_AWS_SECRET_KEY</li></ul><p>Der Befehl vagrant up erstellt ein virtuelles Ubuntu, dass als Tor zu unserem Cluster dienen wird. Während der Provisionierung der virtuellen Maschine erzeugt Ansible das Cluster in AWS. Dieser Prozess ist synchron und kann einige Minuten dauern. Im Anschluss können wir uns unsere neuen EC2-Instanzen in AWS ansehen:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ux4DCmdRmHMmhSJbGhQs"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ux4DCmdRmHMmhSJbGhQs" alt="amazon" width="700" height="230" /></a> <p>Diese EC2-Instanzen wurden erzeugt.</p></div><p>Ansible schreibt zusätzlich die URL des Master-Loadbalancers auf die Konsole:</p>

```
TASK [dcos-cli : This is the URL of your master ELB] ****************
ok: [default] => {
    "msg": "http://dcos-test-ElasticL-***.elb.amazonaws.com"
}
```

<p>Die wichtigsten Einstiegspunkte in euer Cluster sind:</p><ul><li>DC/OS-Dashboard: http://<master_load_balancer>/</li><li>Der Scheduler Marathon: http://<master_load_balancer>/marathon</li><li>Mesos-WebUI: http://<master_load_balancer>/mesos</li></ul><p>Somit können wir uns nun unser brandneues DC/OS-Dashboard ansehen:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tTk1koYQxgeMom50RewG"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tTk1koYQxgeMom50RewG" alt="dcos" width="700" height="391" /></a> <p>Das DC/OS-Dashboard ist unter http://<master_load_balancer> verfügbar.</p></div><p>Da wir noch keine Anwendungen in unserem Cluster installiert haben, stehen sämtliche Ressourcen zur Verfügung.</p><p>Unser DC/OS-Cluster dient als Basisinfrastruktur, auf der wir im Folgenden die noch fehlenden SMACK-Komponenten installieren werden.</p><h2>Schritt 2: <small>sma</small><strong>C</strong><small>k</small> steht für Cassandra</h2><p>Ansible war bereits so nett, die <a href="https://dcos.io/docs/1.7/usage/cli/" target="_blank">DC/OS CLI</a>  in unserer Vagrant-Box zu installieren. Die DC/OS CLI ist ein Kommandozeilenprogramm, mit dem wir Applikationen in unserem Cluster verwalten können. Innerhalb der Vagrant-Box (Zugriff über vagrant ssh) können wir das Cassandra-Mesos-Framework nun mit nur einem einfachen Befehl installieren:</p>

```
dcos package install cassandra
```

<p>Dieser Befehl startet drei Cassandra-Knoten mit Standardeinstellungen bezüglich zugewiesener Prozessoren und Hauptspeicher. Weitere Optionen sind unter <a href="https://dcos.io/docs/1.7/usage/tutorials/cassandra/" target="_blank">https://dcos.io/docs/1.7/usage/tutorials/cassandra/</a>  zu finden.</p><p>Es ist möglich, dass das DC/OS-Dashboard unmittelbar nach diesem Befehl anzeigt, dass Cassandra „unhealthy“ sei. Dieser Zustand ist normal und legt sich nach einigen Minuten. Ihr könnt nun auch sehen, dass erste Ressourcen Eures Cluster nun vergeben sind.</p><p>Die Initialisierung eines Cassandra-Schemas ist leider noch nicht über die DC/OS-CLI möglich. Hierzu müssen wir uns per SSH und dem privaten Schlüssel des Schlüsselpaars „dcos-intro“ auf dem Master-Knoten (nicht Loadbalancer!) einloggen:</p>

```
ssh core@<mesos_master_ip> -i dcos-intro.pem
```

<p>Nun nutzen wir Docker zum Start einer cqlsh-Kommandozeile:</p>

```
docker run -ti cassandra:2.2.5 cqlsh node-0.cassandra.mesos

```

<p>Innerhalb der cqlsh können wir den erforderlichen Keyspace und die einzige Tabelle des Beispiels anlegen:</p>

```
CREATE KEYSPACE IF NOT EXISTS dcos WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3'};
CREATE TABLE IF NOT EXISTS dcos.tweets(date timestamp, text text, PRIMARY KEY (date, text));

```

<p>Ja, es handelt sich um ein Twitter-Beispiel.</p><h2>Schritt 3: <small>smac</small><strong>K</strong> steht für Kafka</h2><p>Analog zu Cassandra können wir auch Kafka über die DC/OS-CLI installieren. Das Kommando</p>

```
dcos package install --package-version=0.9.4.0 kafka
```

<p>installiert den DC/OS-Kafka-Service. Wir verwenden Version 0.9.4.0 des Services, da die mit DC/OS 1.7 ausgerollte Version Probleme in der Interaktion mit den im Beispiel genutzten Frameworks hat.</p><p>Im Gegensatz zu Cassandra führt die Installation des Kafka-Mesos-Dienstes nicht automatisch zum Start von Kafka-Brokern. Hierzu dient folgender DC/OS-CLI-Befehl:</p>

```
dcos kafka broker add 0..2 --port 9092
```

<p>Hierdurch werden drei Broker – broker-0, broker-1 und broker-2 – erzeugt und auf Port 9092 fixiert. Weitere Konfigurationsoptionen sind unter https://docs.mesosphere.com/usage/services/kafka/ verfügbar.</p><p>Broker hinzuzufügen führt noch nicht automatisch zu ihrem Start. Hierfür ist  folgender Befehl zuständig:</p>

```
dcos kafka broker start 0..2
```

<p>Nun nutzen wir die DC/OS-CLI, um ein Kafka-Topic mit dem sehr kreativen Namen „topic“ anzulegen, dass zehn Partitionen und einen Replikationsfaktor von 3 aufweist:</p>

```
dcos kafka topic add topic --partitions 10 --replicas 3
```

<h2>Schritt 4: <strong>S</strong><small>mack</small> steht für Spark</h2><p>Ihr könnt mittlerweile vermutlich raten, wie man Spark installiert. Der Vollständigkeit halber:</p>

```
dcos package install spark
```

<p>Mit diesem Befehl wird Spark im sogenannten <a href="http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode" target="_blank">Cluster-Mode</a>  gestartet. Spark-Jobs werden an den MesosClusterDispatcher geschickt. Dieser Dispatcher verteilt die Jobs und Tasks dynamisch an im Cluster zur Verfügung stehende Ressourcen. Es gibt keine dedizierten Spark-Exekutoren!</p><p>Dieser Schritt bildet den Abschluss der Installation der SMACK-Infrastruktur. Ein Blick auf die Mesos-WebUi offenbart alle laufenden Tasks:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/6CkJxFBNRPOX28vEL2cP"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/6CkJxFBNRPOX28vEL2cP" alt="mesos" width="700" height="649" /></a> <p>Die Mesos-WebUI unter http://<master_load_balancer>/mesos zeigt alle laufenden Tasks</p></div><h2>Das Beispiel</h2><p>Da nun die SMACK-Infrastruktur steht, folgt als nächster logischer Schritt die Ausführung einer SMACK-Anwendung.</p><p>Die Beispielanwendung schiebt kontinuierlich Tweets über Kafka und Spark in die angelegte Cassandra-Tabelle. Sie ist logisch in zwei Teile unterteilt:</p><ul><li>Ingestion (Aufnahme)</li><li>Digestion (Verarbeitung)</li></ul><p>Die folgenden Abschnitte beschäftigen sich mit diesen beiden Themen.</p><h2>Schritt 5: <small>sm</small><strong>A</strong><small>ck</small> steht für Akka – Ingestion</h2><p>Die Anwendung empfängt Tweets über die Twitter4j-Bibliothek und schreibt sie unter Nutzung von Akkas „reactive streams“ und der <a href="https://github.com/akka/reactive-kafka" target="_blank">Kafka-Erweiterung</a>  in das angelegte Kafka-Topic.</p><p>Der Code ist im oben genannten Github-Repository verfügbar. Da es sich um eine Standard-Implementierung ohne große Überraschungen handelt, gehen wir hier nicht viel weiter ins Detail. Viel spannender ist der Deployment-Aspekt. Die Anwendung ist als Docker-Container im öffentlichen Repository <a href="https://hub.docker.com/r/ftrossbach/dcos-intro/" target="_blank">https://hub.docker.com/r/ftrossbach/dcos-intro/</a>  abgelegt.</p><p>Zum Deployment nutzen wir das DC/OS-Scheduling-Framework Marathon und die DC/OS-CLI. Ein Marathon-Deployment wird in JSON beschrieben. Für die Ingestion verwenden wir folgende Konfiguration:</p>

```
{
 "id": "ingestion",
 "container":{
   "docker":{
     "forcePullImage":true,
     "image":"ftrossbach/dcos-intro",
     "network":"BRIDGE"
     "privileged":false
   },
   "type":"DOCKER"
 },
 "cpus":1,
 "mem":2048,
 "env":{
   "KAFKA_HOSTS":"broker-0.kafka.mesos",
   "KAFKA_PORT":"9092",
   "twitter4j.debug":"true",
   "twitter4j.oauth.accessToken":"****",
   "twitter4j.oauth.accessTokenSecret":"****",
   "twitter4j.oauth.consumerKey":"****",
   "twitter4j.oauth.consumerSecret":"****"
 }
}
```

<p>Wir spezifizieren hierbei den zu installierenden Docker-Container, Variablen wie die Anzahl an CPUs oder Hauptspeicher sowie einige Umgebungsvariablen für die Ausführung der Anwendungen.</p><p>Das Deployment wird durch den Befehl</p>

```
dcos marathon app add /vagrant/provisioning/ingestion.json
```

<p> angestoßen. Das JSON muss gültige Twitter-Credentials enthalten.</p><p>Der Status des Deployments kann auf http://<master_load_balancer>/marathon eingesehen werden. Die Mesos-WebUI unter http://<master_load_balancer>/mesos enthält die Informationen ebenfalls. In beiden Oberflächen hat man Zugriff auf Log-Dateien.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/gbuFIFpTM2ozfQvuawdH"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/gbuFIFpTM2ozfQvuawdH" alt="marathon-ingestion" width="636" height="495" /></a> <p>Die Marathon-Übersicht für die Ingestion ist unter http://<master_load_balancer>/marathon/ui/#/apps/ingestion abrufbar.</p></div><p>Das Deployment ist erfolgreich abgeschlossen, wenn die Applikation im Status „running“ ist. Falls dies nicht der Fall ist, prüft Ihr am besten das Error-Log.</p><h2>Schritt 6: <strong>S</strong><small>mack</small> steht schon wieder für Spark – Digestion</h2><p>Im letzten Schritt starten wir nun die Digestion. Die Daten liegen im Kafka und somit auf einem verteilten Message Broker mit extrem hoher Performance bereit. Da Kafka jedoch keinerlei Möglichkeiten für eine effizienten Datenanalyse bietet, müssen wir uns hierfür etwas anderes einfallen lassen.</p><p>Wir nutzen einen Spark-Streaming-Job, der ebenfalls im GitHub-Repository verfügbar ist (de.codecentric.dcos_intro.spark.SparkJob). Um das Beispiel möglichst einfach zu halten, macht dieser Job keinen Gebrauch der vielfältigen Möglichkeiten von Spark. Er schreibt die Daten einfach 1:1 in die angelegte Cassandra-Tabelle.</p><p>Um einen Spark-Job über die DC/OS-CLI zu starten, muss das entsprechende Jar öffentlich verfügbar sein (<a href="https://dcos.io/docs/1.7/usage/tutorials/spark/" target="_blank">https://dcos.io/docs/1.7/usage/tutorials/spark/</a> ). Wir nutzen hierzu AWS-S3. Ist das Jar einmal hochgeladen, kann der Job mit folgendem Kommando gestartet werden:</p>

```
dcos spark run --submit-args=' \ 
--class de.codecentric.dcos_intro.spark.SparkJob \
https://s3.eu-central-1.amazonaws.com/dcos-intro/dcos-intro-assembly-1.0.jar \ 
topic node-0.cassandra.mesos 9042 \ 
broker-0.kafka.mesos:9092'
```

<p>Die WebUI des Spark-Mesos-Dispatchers liefert Auskunft über den Status des Jobs. Sie ist unter<br /> http://<master_load_balancer>/service/spark erreichbar.</p><p>Nachdem der Job angelaufen ist, können wir uns ein weiteres Mal in die Cassandra einloggen und die Tabelle abfragen:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GSoMxeJzTbK5I6xLUJFW"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GSoMxeJzTbK5I6xLUJFW" alt="cassandra-table" width="676" height="374" /></a> <p>Das Ergebnis einer zeitbasierten CQL-Abfrage.</p></div><h2>Zusammenfassung und Ausblick</h2><p>Herzlichen Glückwunsch, Ihr seid nun Herrscher(-innen) über ein voll funktionsfähiges SMACK-Cluster und habt auch schon die erste Anwendung zum Laufen gebracht.</p><p>Dieser Artikel hat Euch durch die hierzu notwendigen Schritte geführt. Das entstandene Cluster eignet sich hervorragend dazu, um die komplette Toolchain zu erproben. Natürlich ist dies nur ein Anfang – die Beispielanwendung kann nicht wirklich als Big- oder Fast-Data-Anwendung bezeichnet werden.</p><p>Vor einem Produktiveinsatz empfehle ich folgende weitere Schritte und Quellen:</p><ul><li>die neue <a href="https://dcos.io" target="blank">DC/OS-Webseite</a> </li><li>die Dokumentation des Schedulers <a href="https://dcos.io/docs/1.7/usage/tutorials/marathon/" target="blank">Marathon</a> </li><li>die Konzepte zur Service-Discovery mit <a href="https://dcos.io/docs/1.7/usage/service-discovery/" target="blank">Mesos DNS</a> </li><li>die Schärfung der Einstellungen für Kafka und Cassandra</li><li>die Einführung von Zugangsbeschränkungen – das genutzte Template erlaubt jedermann Zugriff</li><li>eine Erkundung der Optionen der DC/OS CLI (dcos --help innerhalb der Vagrant-Box)</li><li>die Verschaffung eines Überblicks über die offiziellen DC/OS-Pakete im <a href="https://github.com/mesosphere/universe" target="blank">Mesosphere Universe</a> </li></ul><p>Um den Stack nach der Erprobung mit wenigen Klicks wieder abzureißen, kann die CloudFormation-Konsole in AWS genutzt werden.</p>

Der SMACK-Stack zum Anfassen!

Neues in Apache Kafka 0.10 und Confluent Platform 3.0.0

<p>Version 0.10.0 of the popular distributed streaming platform Apache Kafka saw the introduction of Kafka Streams. In its initial release, the Streams-API enabled stateful and stateless Kafka-to-Kafka message processing using concepts such as map, flatMap, filter or groupBy that many developers are familiar with these days. In Kafka 0.10.1, it started to support “Interactive Queries”, an API that allows querying stateful stream transformations without going through another Kafka topic.</p><p> In this article, we will talk about a specific kind of streaming operation – the joining of streams. We will begin with a brief walkthrough of some core concepts. Then we will take a look at the kinds of joins that the Streams API permits. Following that, we’ll walk through each possible join by looking at the output of an established example. At the end, you should be aware of what kinds of joins are possible in Kafka Streams and where the caveats lie.</p><h2>A brief introduction to some core concepts</h2><p>The central component of Kafka is a distributed pub-sub message broker where producers send messages – key-value pairs – to topics which in turn are polled and read by consumers. Each topic is partitioned and the partitions are distributed among brokers. The <a href="https://kafka.apache.org/documentation/#gettingStarted">excellent Kafka documentation</a>  explains it best.</p><p> There are two main abstractions in the Streams API. A KStream is a stream of key-value pairs – basically a close model of a Kafka topic. The records in a KStream either come directly from a topic or have gone through some kind of transformation – it has for example a filter-method that takes a predicate and returns another KStream that only contains those elements that satisfy the predicate. KStreams are stateless, but they allow for aggregation by turning them into the other core abstraction – KTable – which is often describe as “changelog stream”.</p><p> A KTable statefully holds the latest value for a given message key and reacts automatically to newly incoming messages.<br /> A nice example is perhaps counting visits to a website by unique IP addresses. Let’s assume we have a Kafka topic containing messages of the following type: (key=IP, value=timestamp). A KStream contains all visits by all IPs, even if the IP is recurring. A count on such a KStream sums up all visits to a site including duplicates. A KTable on the other hand only contains the latest message and a count on the KTable represents the number of distinct IP addresses that visited the site.</p><p> KTables and KStreams can also be windowed. Regarding the example, this means we could add a time dimensions to our stateful operations. To enable windowing, Kafka 0.10 changed the Kafka message format to include a timestamp. This timestamp can either be CreateTime or AppendTime. CreateTime is set by the producer and can be be set manually or automatically. AppendTime is the time a message is appended to the log by the broker. Next up: joins.</p><h2>Joins</h2><p>Taking a leaf out of SQLs book, Kafka Streams supports three kinds of joins:</p><ul><li> Inner Joins<ul><li>Emits an output when both input sources have records with the same key.</li></ul></li><li> Outer Joins<ul><li> Emits an output for each record in either input source. If only one source contains a key, the other is null</li></ul></li><li> Left Joins<ul><li> Emits an output for each record in the left or primary input source. If the other source does not have a value for a given key, it is set to null</li></ul></li><p> Another important aspect to consider are the input types. The following table shows which operations are permitted between KStreams and KTables:</p><table><tr><th>Primary Type</th><th>Secondary Type</th><th>Inner Join</th><th>Outer Join</th><th>Left Join</th></tr><tr><td>KStream</td><td>KStream</td><td>Allowed</td><td>Allowed</td><td>Allowed</td></tr><tr><td>KTable</td><td>KTable</td><td>Allowed</td><td>Allowed</td><td>Allowed</td></tr><tr><td>KStream</td><td>KTable</td><td>Not Allowed (Coming 0.10.2)</td><td>Allowed</td><td>Not Allowed</td></tr></table><p>As the table shows, all joins are permitted between equal types. The only permitted inter-type join is a left join between a KStream and a KTable. A possible use case for this join is the matching of incoming streaming data to a KTable that contains some kind of master data, like matching an incoming user id to a more detailed profile.</p><p>In total there are seven possible join types. Let’s look at them in detail. Disclaimer: the table of possible joins and the join semantics are only valid for Kafka 0.10.0 and 0.10.1. They will be improved in Kafka 0.10.2 (see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-77%3A+Improve+Kafka+Streams+Join+Semantics" target="_blank">KIP-77</a> ).</p><h2>Example</h2><p>We are going to use a consistent example to demonstrate the differences in the joins. It is based on the online advertising domain. There is one Kafka topic that contains view events of particular ads and another one that contains click events based on those ads. Views and click share an ID that serves as the key in both topics.</p><p> In the examples, custom set event times provide a convenient way to simulate the timing within the streams. We will look at the following 7 scenarios with ids 0 to 6:</p><ol><li>a click event arrives 1000 ms after the view</li><li>a click event arrives 10,000 ms after the view</li><li>a view event arrives 1000 ms after the view</li><li>there is a view event but no click</li><li>there is click event but no view event</li><li>there are two view events at distinct times, and a click event 1000 ms after the first view. The view events are denoted as 5.1 and 5.2</li><li>there is a view event followed by a click event after 500 ms and another one after 1200 ms. The clicks are denoted as 6.1 and 6.2 in the following.</li></ol><p> This visualization shows these streams:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2gVIcR4QUmjUvP7ladpw"><img width="700" height="155" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2gVIcR4QUmjUvP7ladpw" alt /></a> </p><h2>Inner Join KStream-KStream</h2><p>All KStream-KStream joins are windowed, so the developer has to specify how long that window should be and if the relative order of the elements of both streams matters (ie, happens before/after semantics). The rationale behind that forced windowing is this: a KStream is stateless. To execute a join with acceptable performance, some internal state needs to be kept – otherwise the whole streams would need to be scanned each time a new element arrives. That state contains all elements of the stream within the time window (possible duplicates included). We will use a window of 5000 milliseconds in the following examples.</p><p>An inner join on two streams yields a result if a key appears in the both streams within the window. Applied to the example, this produces the following results:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/az6RPBGuSs6Rf1pSQbw8"><img loading="lazy" width="700" height="223" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/az6RPBGuSs6Rf1pSQbw8" alt /></a> </p><p>Scenarios 0 and 2 appear as expected as the key appears in both streams within 5000 ms, even though they come in different order. Scenario 1 is missing as view and click did not appear within the window. Scenarios 3 and 4 are missing because they do not appear in both streams. Scenarios 5 and 6 appear duplicated as the keys appear twice in the view stream for scenario 5 and in the click stream for scenario 6.</p><p>A variation of this the enforcement of ordering. The developer can specify that a click event can only be joined if it occurs after a view event. This setting would lead to the elimination of scenario 2 in the example.</p><h2>Outer Join KStream-KStream</h2><p>The following will assume the idealistic ordering that each event is processed in the order of its timestamp. In practice, this is not always guaranteed.</p><p>An outer join will emit an output each time an event is processed in either stream. If the window state already contains an element with the same key in the other stream, it will apply the join method to both elements. If not, it will only apply the incoming element.</p><p>The following shows the idealistic result:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/sprjXZNQFuyEwnA0dg38"><img loading="lazy" width="700" height="230" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/sprjXZNQFuyEwnA0dg38" alt /></a> </p><p>For scenario 0, an event is emitted once the view is processed. There is not click yet. When the click arrives, the joined event on view and click is emitted. For case 1, we also get two output events. However, since the events do not occur within the window, neither of these events contains both view and click. Scenario 3 appears in the output without a click, and the equivalent output is emitted for scenario 4. Scenario 5 produces 4 output events as there are two views that are emitted immediately and once again when they are joined against a click. Case 6 only produces 3 events as both clicks can be immediately joined against a view that arrived earlier.</p><p>What happens in real life? Hard to say because the timing is important. In tests, the clicks for scenario 0 were always processed earlier than the views by the streaming application, despite using CreateTime for the message timestamps. This is probably what you should take away from this paragraph: as of Kafka 0.10.1.x, outer joins work on processing time, not message time.</p><p>EDIT: The struck sentence is incorrect. Kafka Streams definitely do not work on processing <strong>time</strong> but event time. However, there are runtime dependencies and processing <strong>order</strong> matters. Thanks to Matthias Sax from Confluent for pointing that out.</p><h2>Left Join KStream-KStream</h2><p>This type of join is processing event-time dependent as well and has some behaviour that you might not expect due to a runtime dependency on processing order.<br /> The left join emits an output event every time an event arrives in the left stream. If an event with the same key has previously arrived in the right stream, it is joined with the one in the primary stream. Otherwise it is set to null. With our examples, that is going to result in a bit of a sad picture as we only have one event where the click arrives before the view and thus has already been processed. This leads to the following result:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/czHvh2XgSPCVEMx8bALp"><img loading="lazy" width="700" height="230" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/czHvh2XgSPCVEMx8bALp" alt /></a> </p><p>Only scenario 2 yields a complete result in the idealistic version, but as expected from a left join, all elements from the left stream show up. This is one of the semantics that will change with Kafka 0.10.2 where “right” messages arriving within the window will cause the emission of a joined event.</p><h2>Inner Join KTable-KTable</h2><p>Now we’re switching from KStreams to KTables. KTables are represented by materializing the incoming data into a local state store, building a table that always contains the latest update for a key. When a new record arrives, it is joined with the other table’s state store. Joins on KTables are not windowed as KTables describe a state, not a stream. The emitted events show state changes within the table.</p><p>It is not easy to figure out what happens here exactly. The following chart represents observations – please correct me if I’m wrong.<br /> EDIT: I have gained a better insight in KTables. The behaviour that tripped me up is caused by caching. It works like this: by default, KTables use a cache for more efficient data processing. Results are only emitted once either the cache is full or a commit interval (default: 30 seconds) has been reached. For a better understanding of the join semantics, disabling caching is very helpful. The following chart shows results for both scenarios:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/oosOSWUDTCO50Ohg09v4"><img loading="lazy" width="700" height="405" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/oosOSWUDTCO50Ohg09v4" alt /></a> </p><p>All the inner join pairs are emitted in both scenarios. Since we’re no longer windowed, even scenario 1 is represented. Empty brackets represent a notification that the state for the keys those scenarios has been updated, but as they could not be joined, the new state is empty/null. With caching enabled all data has been processed before a cache flush happens, so we’re missing events 5.1 and 6.1 from the join output. Another curious thing is that events are duplicated. According to <a href="https://issues.apache.org/jira/browse/KAFKA-4609">KAFKA-4609</a>  this is caused by the fact that both source tables are flushed individually and therefore both trigger the join. This seems to fit as the first set of emissions contains the empty update for scenario 3 whereas the second set contains scenario 4.</p><p>With caching disabled, cache semantics become a lot clearer. Very interesting are scenarios 5 and 6 where the stateful nature of KTables can be observed – view 5.1 is missing as it is updated in the source table by 5.2 wheres we get two joined output for scenario 6 because while click 6.2 updates 6.1, the runtime was able to match 6.1 against view 6 before it was update.</p><h2>Outer Join KTable-KTable</h2><p>Apart from the duplication that occurs in the outer join as well when enabling caching, it behaves pretty much as expected:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/B7dnsJ7EQIG1IiJRDlbQ"><img loading="lazy" width="700" height="418" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/B7dnsJ7EQIG1IiJRDlbQ" alt /></a> </p><p>For the version with caching, the results are the same as with the inner join with the addition of proper data for scenarios 3 and 4 which for scenario 3 contains the lone view and for scenario 4 the lone click.</p><p>The none-caching variant emits an event every time a new message arrives and executes the join if a matching entry is held on the other table.</p><h2>Left Join KTable-KTable</h2><p>Left joins exhibit the same duplication behaviour. Apart from that, they also work the way you’d expect by now:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ji3PTsiJTQmXeqDnjPGB"><img loading="lazy" width="700" height="422" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ji3PTsiJTQmXeqDnjPGB" alt /></a> </p><p>Using Caching we get a click element for case 3 and an empty/null event for scenario 4. The rest of the data is joined completely.<br /> Disabling caching, we can see the semantics much more clearly. They behave as you’d expect a left join to behave.</p><p>This concludes the KTable-KTable section. The big lesson learned here is that the settings for caching play a very big role in the emission of events from a joined KTable. Yet the end result is the same – each joined KTable has the same content after completely processing the sample data, cached or non-cached. But the way we’re getting there is different.</p><h2>Left Join KStream-KTable</h2><p>As of Kafka <= 0.10.1.x,="" this="" is="" the="" only="" type="" of="" stream="" table="" join.="" Its="" semantics="" imply="" that="" an="" incoming="" event="" in="" a="" can="" be="" joined="" against="" table.="" The="" output="" operation="" another="" (and="" not="" table!).="" highly="" timing="" dependent="" –="" example="" with="" views="" and="" clicks="" does="" really="" work="" well="" as="" you’d="" probably="" use="" stream-table="" join="" for="" click="" adserving="" usually="" arrives="" after="" view.="" But="" any="" case,="" please="" consider="" contrieved="" example:="" <img loading="lazy" width="700" height="281" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/xIh6kaXVSzyzK8xjs52M" alt /></p><p>We’re simulating the output of an incoming stream joined against a table of clicks that initially is populated with events 0, 2, 4 and 6.1. Each incoming stream event is joined against that table and the result either contains both view and click or only the view if there is no click to join against. Between views 5.1 and 5.2, click event 5 is registered. While view 5.1 could not be joined against a click, this update means that view 5.2 can be joined. There is no issue with duplication as the result is a stream and not a table.</p><h2>Partitioning and Parallelization</h2><p>If you are familiar with Kafka consumers, you are probably aware of the concept of the consumer groups – Kafka consumption is parallelized by assigning partitions to exactly one consumer in a group of consumers that share the same group id. If you’re using defaults, Kafka itself will handle the distribution and assign the partitions to consumers. This happens in a way that the consumers have little influence over.<br /> With simple consumers, this is quite straightforward. However, what does it mean for a Kafka stream with state and joins? Can partitions still be randomly distributed? No, they cannot. While you can run multiple instances of your streaming application and partitions will be distributed among them, there are requirements that will be checked at startup. Topics that are joined need to be copartitioned. That means that they need to have the same number of partitions. The streaming application will fail if this is not the case. Producers to the topic also need to use the same partitioner although that is something that cannot be verified by the streaming application as the partitioner is a property of the producer. For example, you will not get any join results if you send view event 0 to partition 0 and the corresponding click event to partition 1 even if both partitions are handled by the same instance of the streaming application.</p><h2>Summary and Outlook</h2><p>Kafka Streams is a very interesting API that can handle quite a few use cases in a scalable way. However, some join semantics are a bit weird and might be surprising to developers. An example of this is left and outer join on streams depending on the processing time of the events instead of the event time. Some of those issues are addressed by KIP-77 and are scheduled to be released with Kafka 0.10.2.<br /> The duplicates produced by KTable joins look weird, but they can provide a lot of benefit when used with the “Interactive Query” feature introduced with Kafka 0.10.1 where state stores can be directly queried.</p><p>Kafka’s journey from Pub/Sub broker to distributed streaming platform is well underway and times are very exciting.</p><h2>References</h2><ul><li>The official <a href="http://kafka.apache.org/documentation/streams" target="_blank">Streams documentation</a> </li><li><a href="https://github.com/ftrossbach/crossing-the-streams" target="_blank">A GitHub repository with examples for all joins</a> </li><li><a href="https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics" target="_blank">Kafka Wiki describing current and future join semantics</a> </li><li>Confluent’s <a href="http://docs.confluent.io/3.1.2/streams/index.html" target="_blank">general streams documentation</a> </li><li>First-hand experience with Kafka as part of the <a href="/wissens-hub/blog/smack-stack-trenches" target="_blank">SMACK stack</a> </li></ul></ul>

Crossing the Streams – Joins in Apache Kafka

<p>The <a href="https://www.codecentric.de/leistungen/loesungen/technologie-treibt-veraenderung/big-data/smack/">SMACK stack</a>  is all the rage these days. Instead of just talking about it, this post is going to guide you through the steps for setting up a simple SMACK stack that will enable you to get a hands on experience with the tools.<br /> In the first step, we will setup a DC/OS-managed Mesos datacenter on AWS. This is going to provide the basic infrastructure on which we will then install Kafka, Cassandra and Spark. Then we are going to deploy simple ingestion and digestion jobs that pump data right across the toolchain.</p><p><span></span></p><h2>Step 0: Prerequisites</h2><p>If you want to try the examples yourself, you’ll need the following:</p><ul><li>Vagrant and Virtualbox</li><li>Ansible</li><li>An AWS Account. The AWS resources used in this example exceed the free tier, regular AWS fees apply. You have been warned!</li><li>An EC2 key pair called dcos-intro</li><li>A Twitter access key</li></ul><p>The source code is available at <a href="https://github.com/ftrossbach/intro-to-dcos">https://github.com/ftrossbach/intro-to-dcos</a> .</p><h2>Step 1: <small>s</small><strong>M</strong><small>ack</small> stands for Mesos</h2><p>DC/OS is Mesosphere’s distributed operating system on top of Apache Mesos. Version 1.7 has <a href="https://dcos.io/" target="_blank">just been released as open source</a> . DC/OS provides <a href="https://downloads.dcos.io/dcos/EarlyAccess/aws.html" target="_blank">AWS CloudFormation templates</a>  to set up the datacenter. The following diagram gives a high level overview of the resources that are created for the example in this blog post:</p><p> </p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4qe10tJYSKNCwxRDDZ6s"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/4qe10tJYSKNCwxRDDZ6s" alt="aws" width="700" height="571" /></a> <p>The CloudFormation template creates these AWS resources</p></div><p>In the public subnet that is reachable from the internet, we can see a master node and a public Mesos slave. One master is sufficient as there is no need for high availability (HA). DC/OS also provides templates that provision three masters and thus provide HA capability. The private subnet is only accessible from the public subnet. It contains 5 mesos slaves that will run any application that does not need exposure to the outside world. If you want to know more, you can have a look at <a href="https://github.com/zutherb/terraform-dcos" target="_blank">my colleague Bernd Zuther’s Github repository</a> . Bernd gives a more detailed overview of DC/OS. He also transformed the CloudFormation template into a Terraform template that is more easily readable.</p><p>To get the stack running, we need to check out the project at <a href="https://github.com/ftrossbach/intro-to-dcos" target="_blank">Github</a> . We have also to provide AWS credentials for AWS in form of the environment variables:</p><ul><li>DCOS_INTRO_AWS_REGION</li><li>DCOS_INTRO_AWS_ACCESS_KEY</li><li>DCOS_INTRO_AWS_SECRET_KEY</li></ul><p>Calling vagrant up then sets up a virtual Ubuntu that will serve as our gateway to DC/OS. After the machine is created, the Ansible CloudFormation task takes the template and creates the Stack in AWS. This is happening synchronously, so it will take a couple of minutes. After Ansible is done, we can take a look at our brand new instances:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ux4DCmdRmHMmhSJbGhQs"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ux4DCmdRmHMmhSJbGhQs" alt="amazon" width="700" height="230" /></a> <p>The instances that have been created in AWS as shown in the AWS console</p></div><p>Ansible also prints the URL of the master load balancer:</p>

```
TASK [dcos-cli : This is the URL of your master ELB] ****************
ok: [default] => {
    "msg": "http://dcos-test-ElasticL-***.elb.amazonaws.com"
}
```

<p>The most important URLs for accessing your datacenter are:</p><ul><li>DC/OS dashboard: http://<master_load_balancer>/</li><li>The task scheduling framework Marathon: http://<master_load_balancer>/marathon</li><li>Plain Mesos: http://<master_load_balancer>/mesos</li></ul><p>Let’s take a look at our brand new DC/OS dashboard:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tTk1koYQxgeMom50RewG"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tTk1koYQxgeMom50RewG" alt="dcos" width="700" height="391" /></a> <p>The DC/OS dashboard that is available at http://<master_load_balancer></p></div><p>As we have not yet deployed any applications, all resources are still up for grabs.</p><p>Now that we have Mesos running in form of DC/OS, we are going to install the remaining SMACK infrastructure components.</p><h2>Step 2: <small>sma</small><strong>C</strong><small>k</small> stands for Cassandra</h2><p>Ansible already installed the <a href="https://dcos.io/docs/1.7/usage/cli/" target="_blank">DC/OS CLI</a>  in our Vagrant box. DC/OS CLI is a command line interface to manage applications in the datacenter. From within the Vagrant box (use vagrant ssh), the Cassandra Mesos framework can be installed by issuing one simple shell command:</p>

```
dcos package install cassandra
```

<p>This will start three Cassandra nodes with some defaults regarding memory and disk requirements. Configuration options are described at <a href="https://dcos.io/docs/1.7/usage/tutorials/cassandra/" target="_blank">https://dcos.io/docs/1.7/usage/tutorials/cassandra/</a> .</p><p>If you look at the DC/OS dashboard, you might see that Cassandra reports as unhealthy. It should turn to healthy once all Cassandra nodes are up and running. You can also now see that some resources have been allocated in DC/OS.</p><p>Setting up a Cassandra keyspace is unfortunately not yet possible via the DC/OS CLI. We need to log onto one of our master nodes using SSH and the private key of key pair “dcos-intro”:</p>

```
ssh core@<mesos_master_ip> -i dcos-intro.pem
```

<p>We use docker to start a cqlsh Cassandra command line:</p>

```
docker run -ti cassandra:2.2.5 cqlsh node-0.cassandra.mesos

```

<p>Once in cqlsh, we can create the required keyspace and table:</p>

```
CREATE KEYSPACE IF NOT EXISTS dcos WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3'};
CREATE TABLE IF NOT EXISTS dcos.tweets(date timestamp, text text, PRIMARY KEY (date, text));

```

<p>Yes, this is going to be yet another Twitter-based example.</p><h2>Step 3: <small>smac</small><strong>K</strong> stands for Kafka</h2><p>Similarly to Cassandra, we are also going to use command line tools to set up Kafka.<br /> Issuing the command</p>

```
dcos package install --package-version=0.9.4.0 kafka
```

<p>installs the DC/OS Kafka service. We are using version 0.9.4.0 of this service because the most recent version that was released with DC/OS 1.7 caused some incompatibilities with the frameworks used in the example.</p><p>Compared to Cassandra, this the installation of the framework will not lead to an automatic startup of Kafka brokers. We are going to use the DC/OS Kafka command line to add and start Kafka brokers:</p>

```
dcos kafka broker add 0..2 --port 9092
```

<p>This command adds three brokers – broker-0, broker-1 and broker-2 – that are supposed to run on port 9092. Further configuration options are available at https://docs.mesosphere.com/usage/services/kafka/.<br /> Adding brokers is not the same as starting them, issuing</p>

```
dcos kafka broker start 0..2
```

<p>does that job.</p><p>We also use the CLI to create a Kafka topic very creatively called “topic” with ten partitions and a replication factor of three:</p>

```
dcos kafka topic add topic --partitions 10 --replicas 3
```

<h2>Step 4: <strong>S</strong><small>mack</small> stands for Spark</h2><p>You can probably guess by now how to install Spark, but here it is anyway:</p>

```
dcos package install spark
```

<p>This sets up Spark on Mesos in so called<a href="http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode" target="_blank"> cluster mode</a> . Spark jobs will be submitted to a MesosClusterDispatcher that will distribute Spark jobs on demand to available cluster resources. There are no dedicated Spark executors!</p><p>This completes the installation of the SMACK infrastructure. A look at the Mesos Web UI reveals all running tasks:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/6CkJxFBNRPOX28vEL2cP"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/6CkJxFBNRPOX28vEL2cP" alt="mesos" width="700" height="649" /></a> <p>The Mesos Web UI at http://<master_load_balancer>/mesos shows the running tasks</p></div><h2>The example application</h2><p>By now we have all the infrastructure available to run a SMACK application, so let’s just do that.<br /> The example application streams tweets via Kafka and Spark into Cassandra. It is logically divided into two parts:</p><ul><li>data ingestion</li><li>data digestion</li></ul><p>We’ll see more about how these parts are set up in the following sections.</p><h2>Step 5: <small>sm</small><strong>A</strong><small>ck</small> stands for Akka – data ingestion</h2><p>The application reads tweets using the twitter4j library and pushes them onto Kafka using Akka reactive streams and the <a href="https://github.com/akka/reactive-kafka" target="_blank">reactive kafka</a>  extensions. The code is available in the Github repository and we will not go into detail on this as it is a straightforward Akka stream implementation. Much more interesting is the deployment aspect. The application is packaged as a docker container and deployed to a public docker registry at <a href="https://hub.docker.com/r/ftrossbach/dcos-intro/" target="_blank">https://hub.docker.com/r/ftrossbach/dcos-intro/</a> . To deploy the app in our datacenter, we use DC/OS’ scheduler Marathon and the DC/OS CLI. A new deployment in Marathon is defined in JSON. For the ingestion, we use the following configuration:</p>

```
{
 "id": "ingestion",
 "container":{
   "docker":{
     "forcePullImage":true,
     "image":"ftrossbach/dcos-intro",
     "network":"BRIDGE"
     "privileged":false
   },
   "type":"DOCKER"
 },
 "cpus":1,
 "mem":2048,
 "env":{
   "KAFKA_HOSTS":"broker-0.kafka.mesos",
   "KAFKA_PORT":"9092",
   "twitter4j.debug":"true",
   "twitter4j.oauth.accessToken":"****",
   "twitter4j.oauth.accessTokenSecret":"****",
   "twitter4j.oauth.consumerKey":"****",
   "twitter4j.oauth.consumerSecret":"****"
 }
}
```

<p>In this file, we specify the Docker container we want to deploy, some variables like the number of CPUs or memory, and most importantly some environment variables for the container.<br /> We trigger the deployment by issuing</p>

```
dcos marathon app add /vagrant/provisioning/ingestion.json
```

<p> – you will need to include valid Twitter credentials in that file.</p><p>If you want to check the status of the application, you can take a look into Marathon at http://<master_load_balancer>/marathon. The plain Mesos UI at http://<master_load_balancer>/mesos also has this information. You have access to log files in both UIs.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/gbuFIFpTM2ozfQvuawdH"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/gbuFIFpTM2ozfQvuawdH" alt="marathon-ingestion" width="636" height="495" /></a> <p>The Marathon page for the ingestion application is reachable at http://<master_load_balancer>/marathon/ui/#/apps/ingestion</p></div><p>If the application reports as “running”, you’re fine. If it is stuck in a deploy-wait-cycle, check stderr.</p><h2>Step 6: <strong>S</strong><small>mack</small> stands for Spark (yet again…) – data digestion</h2><p>The final step is running the data digestion. In Kafka, we have a distributed high performance message broker, but we need to do something with the data as Kafka itself does not offer any analytic capabilities whatsoever. We use a Spark streaming job that is also included in the example (de.codecentric.dcos_intro.spark.SparkJob). To keep things uncomplicated, this example does not really make use of the true power of Spark but plainly writes the tweets it reads from Kafka into a Cassandra table.</p><p>To deploy a Spark job using the DC/OS CLI, the jar must be available in a place where DC/OS can access it (<a href="https://dcos.io/docs/1.7/usage/tutorials/spark/" target="_blank">https://dcos.io/docs/1.7/usage/tutorials/spark/</a> ). A public file on AWS S3 will do the trick for us. Once uploaded, the job can be triggered by issuing</p>

```
dcos spark run --submit-args=' \ 
--class de.codecentric.dcos_intro.spark.SparkJob \
https://s3.eu-central-1.amazonaws.com/dcos-intro/dcos-intro-assembly-1.0.jar \ 
topic node-0.cassandra.mesos 9042 \ 
broker-0.kafka.mesos:9092'
```

<p>The Spark Mesos dispatcher has a WebUI that lets us monitor the job status. It is accessible at<br /> http://<master_load_balancer>/service/spark.</p><p>Once the job is running, we can log into Cassandra using the same docker command as above when we created the keyspace and query the tweets table (extract):</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GSoMxeJzTbK5I6xLUJFW"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GSoMxeJzTbK5I6xLUJFW" alt="cassandra-table" width="676" height="374" /></a> <p>This shows the results of a Cassandra query</p></div><h2>Conclusion and outlook</h2><p>Congratulations, you got the SMACK stack up and running and deployed an application.<br /> This post gave you basic steps to setup the stack on AWS and run simple ingestion and digestion jobs. This is great to explore the features of DC/OS and how it enables you to run a fast data application, but of course there are many things left out – our example application is neither fast data nor big data.</p><p>If you want to set up the stack productively, you might also want to:</p><ul><li>check out the brand new <a href="https://dcos.io" target="blank">DC/OS homepage</a> </li><li>read up on <a href="https://dcos.io/docs/1.7/usage/tutorials/marathon/" target="blank">Marathon</a> </li><li>explore service discovery via <a href="https://dcos.io/docs/1.7/usage/service-discovery/" target="blank">Mesos DNS</a> </li><li>think about the number of Kafka and Cassandra nodes as well as their memory requirements</li><li>restrict access to DC/OS – with this template, the master nodes are accessible to everyone on the internet</li><li>explore the options of the DC/OS CLI (dcos --help from within the Vagrant box)</li><li>have a look of other official DC/OS packages in the <a href="https://github.com/mesosphere/universe" target="blank">Mesosphere universe</a> </li></ul><p>In order to clean up, you can delete the Stack on AWS in the CloudFormation console. This will destroy all created resources.</p>

The SMACK stack – hands on!



```
"Databases? Where we're going we don't need databases" – Doc Brown, 1985
```

<p>Well, we’re certainly not there yet, but this article is going to introduce you to a new feature of the popular streaming platform Apache Kafka that can make a dedicated external database redundant for some use cases.</p><p>Kafka 0.10.0 introduced the “Kafka Streams” API – a new Kafka client that enables stateless and stateful processing of incoming messages, with state being stored internally where necessary. In the initial release, state could only be exposed by writing to another Kafka topic. Since Kafka 0.10.1, this internal state can be queried directly. This article introduces the API and talks about the challenges in building a distributed streaming application with interactive queries. It assumes basic knowledge of the <a href="http://kafka.apache.org/documentation/streams" target="_blank">Streams API</a> .</p><h2>Example</h2><p>Let’s consider a simple example that models the tracking of visits to a web page. A topic “visitsTopic” contains Kafka messages that contain key-value pairs in the format <ip,timestamp>. So the key of the message contains an IP address of a visitor and the value contains the timestamp of that visit. This is of course a bit contrived as we would not use one topic per trackable web page, but let’s keep it simple.</p><p>For the sake of this example, we’re interested in three aspects:</p><ul><li>how many times did a user with a given IP visit our page in total?</li><li>how often was it visited by a given IP in the last hour?</li><li>how many times per user session did an IP visit the page?</li></ul><p>This can be achieved with the following topology:</p><p></p><p>There are three state stores:</p><ul><li>“totalVisitCount” contains the total amount of visits per unique IP</li><li>“hourlyVisitCount” contains the number of visits in the last hour</li><li>“sessionVisitCount” contains the count per session (with a new session being started when there is no activity for more than a minute)</li></ul><p>In Kafka 0.10.0, the only option to retrieve that data would have been to materialize it into another Kafka topic. For many use cases, this can be considered quite wasteful. Why do we have to persist the data once again in Kafka if all we want to do is answer a few very simple queries?</p><h3>Interactive Queries to the Rescue</h3><p>As outlined in <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams" target="_blank">KIP-67</a> , interactive queries were designed to give developers access to the internal state that the Streams-API keeps anyway. This is the first bit to take away: interactive queries are not a rich Query-API built on Kafka Streams. They merely make existing internal state accessible to developers.</p><p>The state is exposed by a new method in org.apache.kafka.streams.KafkaStreams. While this client originally mainly contained the capability to start and stop streaming topologies, it has been extended in Kafka 0.10.1 and further with 0.10.2. The entrypoint into querying a local state store is the <a href="https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/KafkaStreams.html#store(java.lang.String,%20org.apache.kafka.streams.state.QueryableStoreType" target="_blank">store</a>  method. Let’s look a bit more closely at its signature:</p><p>public <T> T store(String storeName,<br /> QueryableStoreType<T> queryableStoreType)</p><p>The first parameter is easy, it takes the name of the store that we want to query – “totalVisitCount”, “hourlyVisitCount” or “sessionVisitCount” in our example. It is not the topic name! The second parameter is a bit more intriguing. It declares the type of the provided store. At this point, it’s worth taking a step back to understand what that is about. By default, the Kafka Streams high-level DSL uses RocksDB (http://rocksdb.org/) to store the internal state. This is generally pluggable by the way – you could supply your own StateStoreProvider. RocksDB mainly works in memory but may flush to disk as well. There are three standard types of RocksDB-backed state stores:</p><ul><li>Key-Value based</li><li>Window based</li><li>Session window based (since 0.10.2)</li></ul><p>In our example, “totalVisitCount” is an example of a key-value based state that maps an IP address to a counter. “hourlyVisitCount” is window-based – it stores the count of visits of an IP address as it occurred in a specific time window. “sessionVisitCount” is an example of a session window store. Session windows are a new feature of Kafka 0.10.2 and allow to group repeated occurrences of keys into specific windows that dynamically expand if a new record arrives within a so-called inactivity gap. Simple example: if the inactivity gap is 1 minute, a new session window would be opened if there was no new record for a key for longer than that minute. Two messages within say 20 seconds would belong to the same window.</p><p>Each store type has its specifically tailored API. A key value store enables different types of queries than window stores.</p><p>Accessing a key-value store works like this:</p><p>ReadOnlyKeyValueStore<String, Long> store = streams.store(“visitsTable”,<br /> QueryableStoreTypes.<String, Long>keyValueStore());</p><p>An important aspect of interactive queries is in the name of the return type – they are read-only. There are no inserts, updates, deletes whatsoever. This is a good thing – Kafka topics are your only data source and underlying computations could really get messed up if you were allowed to manipulate data.</p><p>The ReadOnlyKeyValueStore interface does not contain a lot of methods. You can basically query the value of a certain key, the values of a range of keys, all keys and an approximate count of entries. Applied to our example, this store enables you to query for the total count of visits for a given IP, the count for a range of IPs, all IPs and their count and an approximate count of all unique IPs in the store.</p><p>Creating a handle to a windowed store works like this:</p><p>ReadOnlyWindowStore<String, Long> store = streams.store(“hourlyVisitCount”,<br /> QueryableStoreTypes.<String, Long&gtwindowStore());</p><p>This interface is even sparser as it only has one method called fetch that takes a key as well as a “from” and and a “to” timestamp.<br /> This retrieves the aggregated results of the windows that fall into the passed timeframe. The resulting iterator contains KeyValue<Long,V> objects where the long is the starting timestamp of the window and the V the type of the value of that KTable. With regards to the example, this would retrieve the hourly counts for all visits by a given IP starting with the window that contains “timeFrom” and ending with the window containing “timeTo”.</p><p>Session windows stores are retrieved with<br /> ReadOnlySessionStore<String, Long&gt store = streams.store("sessionVisitCount", QueryableStoreTypes.<String, Long>sessionStore());</p><p>The store interface is the simplest of all as it only has one fetch method that takes a key and nothing else. It retrieves the results for all existing session windows at that point in time.</p><p>So this looks easy enough. When running a single instance of the streaming application, all partitions of the topic are handled by that instance and can be queried. However, running a single instance of a consumer is not really what Kafka is about, is it? How do interactive queries work when the partitions of the source topics – and by extension the state – is distributed across instances of your streaming application?</p><h3>Running your application in distributed mode</h3><p>There is no beating around the bush – here be dragons. As mentioned above, interactive queries have not turned Kafka Streams into an almighty query server.<br /> So the bad news is:</p><ul><li>you need an additional layer that glues together your instances</li><li>you need to know which instance(s) would be responsible for a given query</li><li>you need to build it yourself</li></ul><p>Sucks a bit, right? It’s not hard to see where this restriction is coming from, though – building an efficient generalized query facade running in a distributed mode, working for all kinds of data on Kafka is hard when all you can count on is the fact that keys and values are byte arrays containing god knows what. Another main reason for this is that Kafka Streams aims to be completely agnostic to the kind of context it is run in – it does not want to restrict you to certain frameworks. The <a href="https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/" target="_blank">Confluent blog</a>  argues this case very nicely.</p><p> Kafka Streams is not leaving you completely alone with that problem, though.<br /> When you provide the properties for your streaming application, a new one is application.server. This expects a host:port pair that will be published among the instances of your application. This does <strong>not</strong> mean that the Streams API will actually open that port and listen to some sort of request. That is your responsibility and you are completely responsible for communication protocols etc. But it will communicate that endpoint to the other instances via the Kafka protocol, so if you keep your end of the bargain, you can query any instance for metadata and it will provide a comprehensive view. The following illustration demonstrates the setup:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/FZNOaG0JRCqU4L2eQZtS"><img width="651" height="697" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/FZNOaG0JRCqU4L2eQZtS" alt /></a> </p><p>There are two instances of the application, running on 1.2.3.4:42 and 1.2.3.5:4711. A query layer talks to those instances via a user-defined (that means you) protocol. The instances themselves need to run some kind of server that provides endpoints for that protocol. You’re completely free what to use here, there is a lot of choice in the Java ecosystem – Spring MVC, Netty, Akka, Vert.x, you name it). Initially, the query layer needs to know at least one instance by address, but that instance can – if your protocol allows it – pass on the information about the other endpoints. The query layer can ask <strong>any instance</strong> for information about the location of a given key or store.</p><h3>Accessing the metadata</h3><p>So how do we get this metadata on the low level? For this, we return to org.apache.kafka.streams.KafkaStreams. Apart from the method that let’s us access a store, it also provides access to metadata on varying levels. You can simply query all metadata for a streaming application. This will give you an overview of:</p><ul><li>what instances of my application are running where (according to the “application.server” property?</li><li>what state stores are available on those instances?</li><li>what partitions of what topics are handled by an instance?</li></ul><p>In a simple example with only one instance, this metadata looks like this (via its toString):</p><p></p><p>The host info object contains the provided application server values, the three state store names are present and the instance handles partitions 0 and 1 of topic “visitsTopic”. If there were more instances, we’d get all metadata. That metadata is of course a snapshot of the time you call the allMetadata() method – starting or stopping instances can result in partition reassignment.</p><p>The API provides more fine-grained access as well. We can query all the metadata for a given state store, for example. This operation only returns metadata for instances where a store of that name is present. Even more specific are two methods that take the name of a store and a key (and either a Serializer for that key or a StreamPartitioner). This is a very interesting operation as it will return the single metadata for the instance that will hold the data for a key if any data exists, which of course cannot be guaranteed – we won’t know if data is there unless we execute an actual query.</p><h3>Conclusion</h3><p>Interactive queries are a very cool feature that just might make your database redundant one day. Kafka is not the only technology moving in that direction – <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/queryable_state.html" target="_blank">Apache Flink 1.2</a>  introduced a similar feature.<br /> But let’s not get ahead of ourselves – these are early days for this kind of technologies. Interactive queries in Kafka are at the moment only suitable for very simple key-based queries and the need to build your own distributed query layer might put people off. But with an ever-growing Kafka community, there is some real potential. The future is not quite here yet, but interactive queries shows us what it might look like.</p><p>As an entry point for further reading,I recommend reading <a href="https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/" target="_blank">Confluent’s introductory post</a> . Confluent also provides a reference implementation of a query layer.</p>

Interactive Queries in Apache Kafka Streams

<p>Interactive Queries are a fairly new feature of Apache Kafka Streams that provides programmatic access to the internal state held by a streaming application. However, the Kafka API only provides access to the state that is held locally by an instance of the application – there is no global state. Source topic partitions are distributed among instances and while each can provide cluster metadata that tells a caller which instances are responsible for a given key or store, developers must provide a custom RPC layer that glues it all together. While playing around with the API while preparing a blog on <a href="/wissens-hub/blog/interactive-queries-in-apache-kafka-streams">Interactive Queries</a> , I wondered how such a layer could be written in a generic way. This blog describes how I ended up with <a href="https://github.com/ftrossbach/kiqr" target="_blank">KIQR (Kafka Interactive Query Runtime)</a> .</p><p>Disclaimer: This truly is a hobby project and has not been extensively tested at runtime.</p><h2>First steps</h2><p>After looking at the default APIs on the KafkaStreams client class, I realized I had to account for two types of queries:</p><ul><li>key-based queries that would only be routed to one instance in the cluster based on the key</li><li>scatter-gather queries that would be routed to all instances that held data for a given store (by name) and aggregate the results</li></ul><p>Both types involve querying at least one instance. Any instance of a Kafka Streams application can be used to obtain cluster wide metadata that tell us which instance holds what information. But once we know the “where”, how do we get there? Of course we could just communicate via HTTP, but that doesn’t sound that appealing for “internal” queries.<br /> After having heard a lot about <a href="http://vertx.io/">Eclipse Vert.x</a>  from my colleague and <a href="http://vertx.io/blog/scala-is-here/" target="_blank">Vert.x committer Jochen Mader</a> , I thought it might be a good fit. I started reading the Vert.x documentation, and I really liked what I saw.</p><h2>What is Vert.x</h2><p>Vert.x is an event-driven non-blocking application platform. It enables you to write concurrent code without having to think too much about concurrency itself, so you can focus on your business logic instead of threads and synchronization. A key abstraction is the <a href="http://vertx.io/docs/vertx-core/java/#_verticles" target="_blank">Verticle</a> , which works similarly to actors in the actor model (it’s not a perfect match, but close enough). As I was familiar with Akka already, making the leap to Vert.x was actually quite easy. There are some other nice features as well – Vert.x is polyglot, so you can write your components in different languages. It also integrates very well with OSGi. And the list is even longer – by now I’m really excited about Vert.x!</p><p>Components in a Vert.x application communicate via simple String addresses on an <a href="http://vertx.io/docs/vertx-core/java/#event_bus" target="_blank">event bus</a> , and this is the killer feature for KIQR’s use case. It is very simple to run Vert.x in cluster mode, turning the event bus into a distributed event bus without having to change any code. After trying it out with very simple hello world example, this looked capable of handling KIQR’s requirements for internal communication. There are actually <a href="http://vertx.io/docs/#clustering" target="_blank">four libraries</a>  that can be used to run Vert.x in cluster mode (as of Vert.x 3.4.0). The two stable ones are Hazelcast and Apache Ignite. Infinispan and Apache Zookeeper are in technical preview. I settled on Hazelcast as it was the only stable option at the time when I started.<br /> Perfect – transparent communication between the instances is delegated to Vert.x.</p><h3>Componentizing the runtime</h3><p>The event bus sits in the middle, that much is clear. Now what kinds of components do we attach to the bus? I settled on these logical components:</p><ul><li>query verticles for the low-level query operations directly on the KafkaStreams client<ul><li>one for each query operation, potentially multiple ones per store type</li></ul></li><li>query facades that first find out which instances need to be queried, asynchronously execute the query and aggregate the results if necessary<ul><li>also one for each query operation</li></ul></li></ul><p>We definitely need to run the query verticles on every instance that we want to query, so they’ll have to listen to messages on the event bus. But how can we make the correlation between event bus addresses and KafkaStreams metadata? Since Kafka 0.10.1, the Streams API contains a new parameter called application.server that is published among all instances of a streaming application via the Kafka protocol.<br /> As the Vert.x event bus only uses Strings as addresses, I had the idea that I could use that field not to publish a <host:port> pair as intended, but to use a unique identifier as host and listen to that identifier on the event bus. UUIDs make good identifiers in this case.</p><p>The query facades do not actually need to be deployed on every instances as well as they’ll delegate queries to the responsible query verticle, but for simplicity, better load distribution and reduced latency, it won’t hurt to have them run on each instance as well. Facades for the same query type will share the same static address across instances as it really doesn’t matter which instance serves a request. Vert.x will prefer a local one. A query facade asks the KafkaStreams client for metadata, infers the id of the query verticle and issues a request for that verticle on the event bus. The following diagram shows the setup:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ihV7qrTRRmaOod5mKLYZ"><img width="700" height="271" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ihV7qrTRRmaOod5mKLYZ" alt /></a> </p><p>That covers the basic blocks. What’s still missing is a component that opens an interface to the outside world. While other options are conceivable, HTTP is a good start. Vert.x makes it very easy to start a HTTP server and provide a REST-API. That API of course only allows GET requests because Interactive Queries are read-only. Let’s look at the communication flow for a key-value query. All communication between component uses the event bus:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HW9kPhyiSnO5WJ5FREpw"><img loading="lazy" width="417" height="386" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HW9kPhyiSnO5WJ5FREpw" alt /></a> </p><p>As the diagram indicates, this is all as non-blocking as it can be on the server side.</p><p>The following diagram shows an overview of all the verticles that are running in a single KIQR instance:</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tRqDTnPwTpCqPae8rADB"><img loading="lazy" width="700" height="328" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tRqDTnPwTpCqPae8rADB" alt /></a> </p><h3>Serialization</h3><p>As we’re definitely going to have communication between JVMs and wire transfers both within the Vert.X cluster and in communication with clients, we need to think about serialization.<br /> In Kafka, messages are little more than key-value pairs of byte arrays. Producers and consumers need to have a contract on the serialization format. This is informal – Kafka Brokers simply do not care about message contents. That’s why the Producer/Consumer-API heavily rely on <a href="https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/common/serialization/Serde.html" target="_blank">Serdes</a>  (Serializer/Deserializers). As we need those anyway to run Kafka Producers and Streams, we can just go on and use them for all other wire transfers as well – no need to reinvent the wheel. KIQRs runtime will directly serialize any key or value it reads from an interactive query. It will then be encoded as Base64 string. KIQR itself remains as agnostic to message contents as Kafka itself is.</p><p> Serialization on the Vert.x event bus is a different topic altogether. For each message sent over the event bus, Vert.x must be aware of a message codec for that type – even if the message is transmitted within the same JVM. This is a safeguard as the sender is not aware if the receiver is running on the same or a different node. If it is JVM internal, it will not be serialized, but if it needs to be serialized after all, Vert.x knows what to do. KIQR uses simple POJOs that can be easily converted to JSON. Problem solved. This probably could be more efficient, but hey, early days.</p><h3>Server-side example</h3><p>So how can we deploy a Kafka Streams application in with KIQR? First thing you need is a Vertx object. In the simplest case without distribution, this is created by a simple Vertx vertx = Vertx.vertx();. The distributed case involves setting up a cluster manager as per the following example using Hazelcast:</p><p></p><p>In the default, this uses UDP broadcasts as cluster discovery mechanism. If that is not available in your environment (e.g. AWS), please check <a href="http://vertx.io/docs/vertx-hazelcast/java/" target="_blank">the docs</a> .</p><p>Once we got a Vertx object, we can deploy the KIQR verticles. A streaming topology can be started like this:</p><p></p><p>This starts the streaming application with a HTTP server listening on port 4711.</p><h3>Rest-API</h3><p>KIQR supports all standard store operations available in the High Level Streams DSL as of Kafka 0.10.2.0. This is the mapping of endpoints to methods:</p><ul><li>Key-Value queries:<ul><li>/api/v1/kv/{store}/values/{b64 encoded serialized key}?keySerde=<class name>&valueSerde=<class name><ul><li>Maps to <a href="https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#get(K)" target="_blank">org.apache.kafka.streams.state.ReadOnlyKeyValueStore#get</a> </li></ul></li><li>/api/v1/kv/{store}?keySerde=<class name>&valueSerde=<class name><ul><li>Maps to <a href="https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#all()" target="_blank">org.apache.kafka.streams.state.ReadOnlyKeyValueStore#all</a> </li></ul></li><li>/api/v1/kv/{store}?keySerde=<class name>&valueSerde=<class name>&from=<b64 encoded key>&to=<b64 encoded key><ul><li><a href="https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#range(K,%20K)" target="_blank">Maps to org.apache.kafka.streams.state.ReadOnlyKeyValueStore#range</a> </li></ul></li><li>/api/v1/kv/{store}/count<ul><li>Maps to <a href="https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#approximateNumEntries()" target="_blank">org.apache.kafka.streams.state.ReadOnlyKeyValueStore#approximateNumEntries</a> </li></ul></li></ul></li><li>Window queries:<ul><li>/api/v1/window/{store}/{b64 encoded serialized key}?keySerde=<class name>&valueSerde=<class name>&from=<b64 long timestamp>&to=<long timestamp><ul><li>Maps to <a href="https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/state/ReadOnlyWindowStore.html#fetch(K,%20long,%20long)" target="_blank">org.apache.kafka.streams.state.ReadOnlyWindowStore#fetch</a> </li></ul></li><li>/api/v1/session/{store}/{b64 encoded serialized key}?keySerde=<class name>&valueSerde=<class name><ul><li>Maps to <a href="https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/state/ReadOnlySessionStore.html#fetch(K)" target="_blank">org.apache.kafka.streams.state.ReadOnlySessionStore#fetch</a> </li></ul></li></ul></li></ul><h3>Clients</h3><p>You can use the REST API with any client of course, but its URIs contain Base64 encoded serialized keys and the responses also contain serialized values, so a client that handles all that serialization and deserialization sounded like a good idea. The first draft of KIQR contains a REST client based on Apache HttpComponents. The list of dependencies is intentionally kept simple and is restricted to</p><ul><li>Fluent-HC from HttpComponents</li><li>Jackson for a bit of JSON handling</li><li> Kafka Streams (for the Serde interface and the default Serdes)</li></ul><p>Plus transitive dependencies, of course. The clients are blocking for the moment, which marks a bit of a step back from all this non-blocking Vert.x code. But non-blocking clients are definitely on the road map. The clients are written in a way that lets you use the actual types of your keys and values. It will use the provided Serdes to handle wire transfers.</p><p>There is a generic client whose parameters map closely to the REST API:</p><p></p><p>There is also a specific client that let’s you set types, serdes and store name once in the constructor so you don’t have to bother with them each time:</p><p></p><p>This API is probably more enjoyable to use.</p><h3>Caveats and restrictions</h3><p>As mentioned earlier, KIQR is a hobby project. It has not been used in any actually real-world scenario so far. Some other caveats and restrictions are:</p><ul><li>not very well integrationally tested yet, especially not for high volumes</li><li>not highly available in the sense that when the streams app is rebalancing, we cannot execute queries</li><li>No streaming of large results – if you query too much data, you’ll get large results and might run into timeouts</li><li>highly unstable API and implementation, things will change</li><li>you are responsible to know the names of the state stores and types of your keys and values in Kafka. There is<br /> no way to infer them at runtime.</li><li>Java 8 and Kafka Streams 0.10.2 required</li></ul><h2>Conclusion & resources</h2><p>I had a lot of fun building this proof of concept and learned a lot about Vert.x and Interactive Queries on the way. I’d be very happy for feedback.</p><ul><li><a href="https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/" target="_blank">Confluent’s introductory blog for interactive queries</a> </li><li><a href="https://github.com/ftrossbach/kiqr" target="_blank">KIQR source code</a> </li><li>Confluent’s reference implementation</li><li><a href="/wissens-hub/blog/interactive-queries-in-apache-kafka-streams" target="_blank">My article on Interactive Queries</a> </li></ul>

Building a distributed Runtime for Interactive Queries in Apache Kafka with Vert.x

<p>Eight years after its inception, Project Jigsaw – the modularization of the Java platform and introduction of a general module system – is on track to be included in Java 9. The target release has changed over the years from Java 7 via Java 8 to Java 9. The scope has changed a couple of times as well. Now, it really seems as if Jigsaw is getting ready for prime time as it was heavily featured in Oracle’s keynote for JavaOne 2015, with <a href="http://openjdk.java.net/projects/jigsaw/">quite a few sessions focussing on the topic</a> . What does this mean for you? What is Project Jigsaw and how can you use it?<br /> <span><br /> This two-part blog series aims to give you a quick introduction to the module system and lots of practical code examples to demonstrate Jigsaw’s behaviour. In this first part, we’ll talk a bit about what the module system is, how the JDK has been modularized and we’ll demonstrate how the compiler and runtime behave in certain situations.</span></p><h3>So what is a module?</h3><p>Describing a module is actually quite simple – it is a unit of software that declares the answers to three questions about itself in a file named module-info.java:</p><ul><li>What is its name?</li><li>What does it export?</li><li>What does it require?</li></ul><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2hV8SesfTJigzsIOQ4cl"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2hV8SesfTJigzsIOQ4cl" alt="module" width="250" height="156" /></a> <p>A simple module</p></div><p>The answer to the first question is easy, (almost) each module is given a name. This name should be following something similar to the package convention, e.g. de.codecentric.mymodule, to avoid conflicts.</p><p>To answer the second question, modules provide a list of all the <strong>packages</strong> of this particular module that are considered public API and thus usable by other modules. If a class is not in an exported package, no one outside of your module can access it – even if it is public.</p><p>The third question is answered by a list of <strong>modules</strong> that this module depends on – all public types that those modules <strong>export</strong> are accessible by the module in question. The Jigsaw team is trying to establish the phrase “<strong>reading</strong> another module”.</p><p>This is a major shift from the status quo. Up to Java 8, every public type on your classpath is accessible by every other type. With Jigsaw, accessibility of Java types evolves from</p><ul><li>public</li><li>private</li><li>default</li><li>protected</li></ul><p>to</p><ul><li>public to everyone who reads this module (exports)</li><li>public to some modules that read this module (exports to, this will be covered in part two)</li><li>public to every other class within the module itself</li><li>private</li><li><default></li><li>protected</li></ul><h3>The modularized JDK</h3><p>Module dependencies must form an acyclic graph, forbidding circular dependencies. To adhere to this principle, it was a major task for the Jigsaw team to modularize the Java Runtime which was reportedly full of circular and unintuitive dependencies. They came up with is this graph:</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/dn1tTxxfSz2iFfGTGVE2"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/dn1tTxxfSz2iFfGTGVE2" alt="jdk-tr" width="250" height="85" /></a> <p>https://bugs.openjdk.java.net/secure/attachment/21573/jdk-tr.png</p></div><p>At the bottom of the graph resides java.base. This is the only module that only has inbound edges. Every module you create <strong>reads</strong> java.base whether you declare it or not – similar to the implied extension of java.lang.Object. java.base exports packages such as java.lang, java.util, java.math etc.</p><p>The modularization of the JDK means that you can now specify which modules of the Java Runtime you want to use – your application does not need to run in an environment that supports Swing or Corba if you do not read the java.desktop or java.corba modules. How to create such a stripped down environment will be shown in part two.<br /> But enough with the dry stuff…</p><h3>Let’s get hacking</h3><p>All of the code that follows is available <a href="https://github.com/ftrossbach/IntroToJigsaw" target="_blank">here</a> , including the shell scripts for compiling, packaging and running an example.</p><p>My basic use case is very simple. I have a module named de.codecentric.zipvalidator that performs some kind of zip code validation. This module is read by the module de.codecentric.addresschecker (which could check more things than the zip code, but doesn’t to keep things lean).</p><p>The zipvalidator is specified by the following module-info.java:</p>

```

module de.codecentric.zipvalidator{
    exports de.codecentric.zipvalidator.api;        
}

```

<p>So this module exports the de.codecentric.zipvalidator.api package and does not read any other module (except for java.base). This module is read by the addresschecker:</p>

```

module de.codecentric.addresschecker{
    exports de.codecentric.addresschecker.api;
    requires de.codecentric.zipvalidator;
}

```

<p>The overall file system structure is</p>

```

two-modules-ok/
├── de.codecentric.addresschecker
│   ├── de
│   │   └── codecentric
│   │       └── addresschecker
│   │           ├── api
│   │           │   ├── AddressChecker.java
│   │           │   └── Run.java
│   │           └── internal
│   │               └── AddressCheckerImpl.java
│   └── module-info.java
├── de.codecentric.zipvalidator
│   ├── de
│   │   └── codecentric
│   │       └── zipvalidator
│   │           ├── api
│   │           │   ├── ZipCodeValidator.java
│   │           │   └── ZipCodeValidatorFactory.java
│   │           ├── internal
│   │           │   └── ZipCodeValidatorImpl.java
│   │           └── model
│   └── module-info.java

```

<p>By convention, modules are placed in a folder that shares the same name as the module.</p><p>For the first example, everything looks fine – we’re extremely well-behaved and only access the ZipCodeValidator and ZipCodeValidatorFactory from the exported package in our AddressCheckerImpl class:</p><div>

```java
public class AddressCheckerImpl implements AddressChecker {
    @Override
    public boolean checkZipCode(String zipCode) {
        return ZipCodeValidatorFactory.getInstance().zipCodeIsValid(zipCode);
    }
}
```

</div><p>So let’s fire up javac and generate some bytecode. To compile the zipvalidator (which we need to do first of course as the addresschecker reads the zipvalidator), we issue</p><div>

```bash
javac -d de.codecentric.zipvalidator \
$(find de.codecentric.zipvalidator -name "*.java")
```

</div><p>This looks familiar – there is no mention of modules yet as the zipvalidator does not depend on any custom module. The find just helps us with listing .java files within the given folder.<br /> But how do we tell javac about our module structure when we want to compile? For this, Jigsaw introduces the switch -modulepath or -mp.<br /> To compile the addresschecker, we use the following command:</p><div>

```bash
javac -modulepath . -d de.codecentric.addresschecker \
$(find de.codecentric.addresschecker -name "*.java")
```

</div><p>By using the modulepath, we tell javac where to find the compiled modules (in this case, this is .), so this is somewhat similar to the classpath switch.</p><p>Compiling multiple modules separately seems like a hassle though – we can make use of another switch called -modulesourcepath to compile multiple modules at once:</p><div>

```bash
javac -d . -modulesourcepath . $(find . -name "*.java")
```

</div><p>This searches all subdirectories of . for module folders and compiles all java files within.</p><p>Once we have compiled everything, we naturally want to give it a go:</p><div>

```bash
java -mp . -m de.codecentric.addresschecker/de.codecentric.addresschecker.api.Run 76185
```

</div><p>Once again we specify a modulepath so that the JVM knows where to find the compiled modules. We also specify a main class (and a parameter).</p><p>Hooray, the output is</p>

```
76185 is a valid zip code
```

<p>Lucky us!</p><h3>Modular Jars</h3><p>Of course, in the Java world we are used to receive and deliver our bytecode in jar files. Jigsaw introduces the concept of the <strong>modular jar</strong>. A modular jar is very similar to a regular jar, but it also contains a compiled module-info.class. Providing they’re compiled for the right target version, these jars will be downward compatible. module-info.java is not a valid type name, so a compiled module-info.class will be ignored by older JVMs.<br /> To build a jar for the zipvalidator, we issue</p><div>

```bash
jar --create --file bin/zipvalidator.jar \
--module-version=1.0 -C de.codecentric.zipvalidator .
```

</div><p>We specify an output file, a version (though there is no specific notion of using multiple versions of a module in Jigsaw at runtime) and the module to package.</p><p>As the addresschecker also has a main class, we can specify this as well:</p><div>

```bash
jar --create --file=bin/addresschecker.jar --module-version=1.0 \
--main-class=de.codecentric.addresschecker.api.Run \
-C de.codecentric.addresschecker .
```

</div><p>The main class is not being specified in module-info.java as one might expect (and as was initially planned by the Jigsaw team) but written into the Manifest as usual.<br /> Running this example with</p><div>

```bash
java -mp bin -m de.codecentric.addresschecker 76185
```

</div><p>results in the same answer as before. We again specify the modulepath which in this example is the bin folder which we wrote our jars to. We do not need to specify a main class as the Manifest of addresschecker.jar already contains this information. Providing the module name to the -m switch is sufficient.</p><p>Until now, everything has been well and dandy. Now we start tinkering with the modules a bit to see how Jigsaw behaves at compile and runtime when you’re not being a good {girl|boy}.</p><h3>Using non exported types</h3><p>In this example, we’ll see what happens when we access a type from another module that we shouldn’t use.</p><p>Since we’re bored of this factory thing in AddressCheckerImpl, we’ll change the implementation to</p><div>

```java
return new ZipCodeValidatorImpl().zipCodeIsValid(zipCode);
```

</div><p>Trying to compile this results in a well-earned</p><div>

```bash
error: ZipCodeValidatorImpl is not visible because 
package de.codecentric.zipvalidator.internal is not visible
```

</div><p>So directly using non-exported types fails at compile time.</p><p>Feeling smart, we try to be a bit more sneaky and use reflection.</p><div>

```java
ClassLoader classLoader = AddressCheckerImpl.class.getClassLoader();
try {
    Class aClass = classLoader.loadClass("de.[..].internal.ZipCodeValidatorImpl");
    return ((ZipCodeValidator)aClass.newInstance()).zipCodeIsValid(zipCode);
} catch (Exception e) {
    throw new  RuntimeException(e);
}
```

</div><p>This compiles just fine, so let’s run it. Ah, Jigsaw is not fooled that easily:</p><div>

```bash
java.lang.IllegalAccessException:
class de.codecentric.addresschecker.internal.AddressCheckerImpl 
(in module de.codecentric.addresschecker) cannot access class [..].internal.ZipCodeValidatorImpl 
(in module de.codecentric.zipvalidator) because module
de.codecentric.zipvalidator does not export package
de.codecentric.zipvalidator.internal to module
de.codecentric.addresschecker
```

</div><p>So Jigsaw does not only include compile-time checks, but also runtime checks! And it’s very explicit about what we did wrong, too.</p><h3>Circular dependencies</h3><p>In the next case, we have suddenly realized that the addresschecker module contains a class in its API that the zipvalidator would very much like to use. Since we’re lazy, instead of refactoring the class to another module, we declare a dependency to the addresschecker:</p>

```

module de.codecentric.zipvalidator{
        requires de.codecentric.addresschecker;
        exports de.codecentric.zipvalidator.api;

}

```

<p>As cyclic dependencies are not allowed by definition, the compiler stands in the way of our happiness (but for the common good):</p><div>

```bash
./de.codecentric.zipvalidator/module-info.java:2: 
error: cyclic dependence involving de.codecentric.addresschecker
```

</div><p>We cannot do this and are made aware of the problem early at compile time.</p><h3>Implied readability</h3><p>To provide some more functionality, we decide to extend the zipvalidator by introducing a new module de.codecentric.zipvalidator.model that contains some kind of model for the validation result instead of using a boring boolean. The new file structure is shown here:</p>

```

three-modules-ok/
├── de.codecentric.addresschecker
│   ├── de
│   │   └── codecentric
│   │       └── addresschecker
│   │           ├── api
│   │           │   ├── AddressChecker.java
│   │           │   └── Run.java
│   │           └── internal
│   │               └── AddressCheckerImpl.java
│   └── module-info.java
├── de.codecentric.zipvalidator
│   ├── de
│   │   └── codecentric
│   │       └── zipvalidator
│   │           ├── api
│   │           │   ├── ZipCodeValidator.java
│   │           │   └── ZipCodeValidatorFactory.java
│   │           └── internal
│   │               └── ZipCodeValidatorImpl.java
│   └── module-info.java
├── de.codecentric.zipvalidator.model
│   ├── de
│   │   └── codecentric
│   │       └── zipvalidator
│   │           └── model
│   │               └── api
│   │                   └── ZipCodeValidationResult.java
│   └── module-info.java


```

<p>ZipCodeValidationResult is a simple enum that has instances such as “too short”, “too long” etc.</p><p>The module-info.java has been extended to this</p>

```

module de.codecentric.zipvalidator{
       exports de.codecentric.zipvalidator.api;
       requires de.codecentric.zipvalidator.model;
}

```

<p>Our implementation of ZipCodeValidator now looks like this</p><div>

```java
@Override
public <strong>ZipCodeValidationResult</strong> zipCodeIsValid(String zipCode) {
   if (zipCode == null) {
       return ZipCodeValidationResult.ZIP_CODE_NULL;
[snip]
   } else {
       return ZipCodeValidationResult.OK;
   }
}
```

</div><p>The addresschecker module has been adapted to work with this enum as return type as well, so we’re good to go, right? Wrong! Compiling results in</p><div>

```bash
./de.codecentric.addresschecker/de/[..]/internal/AddressCheckerImpl.java:5: 
error: ZipCodeValidationResult is not visible because package
de.codecentric.zipvalidator.model.api is not visible
```

</div><p>There is an error in the compilation of the addresschecker – the zipvalidator uses exported types from the zipvalidator <strong>model</strong> in its public API. Since the addresschecker does not read this module, it cannot access this type.<br /> There are two solutions for this. The obvious one is adding a read edge from the addresschecker to the zipvalidator model. This is however a slippery slope – why would we need to declare this dependency if we only need it for using the zipvalidator? Shouldn’t the zipvalidator ensure that we can access all required modules? It should and it can – welcome to <strong>implied readability</strong>. By adding the keyword public to the requires definition, we tell all client modules that they also need to read another module. As an example, this is the updated module-info.java of the zipvalidator:</p>

```

module de.codecentric.zipvalidator{
       exports de.codecentric.zipvalidator.api;
       requires public de.codecentric.zipvalidator.model;
}

```

<p>The public keyword tells all modules that read the zipvalidator that they also need to read the zipvalidator <strong>model</strong>. This is a change of what you’re used to with the classpath – you cannot rely on a Maven POM for example to ensure your dependencies are also available to any client, you have to specify them explicitly if they are part of your public API. This is a very nice model – if you use a dependency only internally, why should any of your clients be bothered with them? And if you use it externally, you should be open about it, too.</p><h3>Summary and Outlook</h3><p>This concludes part one. We talked about the three questions each module answers and the modularization of the Java Runtime. We proceeded with an example where we compiled, ran and packaged a simple Java application consisting of two modules. Then, we started tinkering with the working example to see how the module system behaves when its rules are violated. An extension of functionality then introduced a third module and the concept of implied readability.</p><p>The <a href="/wissens-hub/blog/first-steps-with-java9-jigsaw-part-2">next part</a>  will cover the following topics:</p><ul><li>How does Jigsaw behave when there are multiple modules with the same name on the modulepath?</li><li>What happens when there are modules on the modulepath that have different names but export the same packages?</li><li>How do you work with legacy dependencies that are not modularized?</li><li>How do you create your own stripped-down runtime image?</li></ul><p>For more information about Jigsaw I recommend <a href="http://openjdk.java.net/projects/jigsaw/" target="_blank">the Jigsaw Project home page</a> , especially the <a href="http://openjdk.java.net/projects/jigsaw/" target="_blank">slides and videos</a>  from this year’s JavaOne sessions and project lead Mark Reinhold’s essay <a href="http://openjdk.java.net/projects/jigsaw/spec/sotms/">State of the Module System</a> .</p>

First steps with Java 9 and Project Jigsaw – Part 1

<p>This is part 2 of a series that aims to get you started with project Jigsaw. In <a href="/wissens-hub/blog/first-steps-with-java9-jigsaw-part-1">part 1</a> , we briefly talked about the definition of a module and how the Java Runtime was modularized. We then proceeded to a simple example that demonstrated how to (and not to) compile, package and run a modular application.<br /> <span><br /> In this post, we try to answer the following questions:</span></p><ul><li>Can I put a restriction on which modules can read an exported package?</li><li>What about having different versions of a module on the modulepath?</li><li>How does Jigsaw interoperate with non-modular legacy code?</li><li>How do I build my own Java runtime image?</li></ul><p>We’ll take the example from part 1 as a base and continue to play with it. The code remains available <a href="https://github.com/ftrossbach/IntroToJigsaw" target="_blank">here</a> .</p><h2>Restricting readability to specific modules</h2><p>In <a href="/wissens-hub/blog/first-steps-with-java9-jigsaw-part-1">part one</a> , we talked about how Jigsaw evolves Java accessiblity. One accessibility level that was mentioned but not elaborated on was “public to some modules that read this module”. In this case, we are able to restrict which modules are allowed to read our exported packages. So if the developers of de.codecentric.zipvalidator really dislike the team behind de.codecentric.nastymodule, they can change their module-info.java to this:</p>

```

module de.codecentric.zipvalidator{

    exports de.codecentric.zipvalidator.api 
        to de.codecentric.addresschecker;
}
```

<p>This allows only the addresschecker to access the zipvalidator API. The specification happens on a package level, so you are perfectly able to restrict access of some package but allow full access for others. This is known as <strong>qualified exports</strong>. If de.codecentric.nastymodule tries to access any type from de.codecentric.zipvalidator.api, there is a compilation error:</p><div>

```bash
./de.cc.nastymodule/de/cc/nastymodule/internal/AddressCheckerImpl.java:4: 
error: ZipCodeValidatorFactory is not visible 
       because package de.cc.zipvalidator.api is not visible
```

</div><p>Note that there is no complaint about module-info.java as the zipvalidator could actually export visible packages to the nastymodule. Qualified exports can for example be used when you want to modularize your application internally, but do not want to share the internal modules’ export packages with clients.</p><h3>Conflicting module versions</h3><p>A common scenario is ending up having different versions of a library in the same application via transitive dependencies, so we could find ourselves in a situation in which a module is on the modulepath twice. Two scenarios come to mind:</p><ul><li>The modules are available at compile time in different folders or modular jars, but still share the same module name</li><li>The different versions of a module have different module names</li></ul><p>Let’s try to compile the application in the first scenario. The zipvalidator has been copied:</p>

```

two-modules-multiple-versions
├── de.codecentric.addresschecker
│   ├── de
│   │   └── codecentric
│   │       └── addresschecker
│   │           ├── api
│   │           │   ├── AddressChecker.java
│   │           │   └── Run.java
│   │           └── internal
│   │               └── AddressCheckerImpl.java
│   └── module-info.java
├── de.codecentric.zipvalidator.v1
│   ├── de
│   │   └── codecentric
│   │       └── zipvalidator
│   │           ├── api
│   │           │   ├── ZipCodeValidator.java
│   │           │   └── ZipCodeValidatorFactory.java
│   │           ├── internal
│   │           │   └── ZipCodeValidatorImpl.java
│   │           └── model
│   └── module-info.java
├── de.codecentric.zipvalidator.v2
│   ├── de
│   │   └── codecentric
│   │       └── zipvalidator
│   │           ├── api
│   │           │   ├── ZipCodeValidator.java
│   │           │   └── ZipCodeValidatorFactory.java
│   │           ├── internal
│   │           │   └── ZipCodeValidatorImpl.java
│   │           └── model
│   └── module-info.java

```

<p>The duplicated modules reside in different folders, but the module name remains unchanged. What has Jigsaw to say about that during compilation?</p><div>

```bash
./de.codecentric.zipvalidator.v2/module-info.java:1: 
error: duplicate module: de.codecentric.zipvalidator
```

</div><p>Ok, so we’re not getting away with this. Jigsaw produces compile error when there are two modules with the same name on the modulepath.</p><p>What about case two? The directory structure remains the same, but now both zipvalidators get different names (de.codecentric.zipvalidator.v{1|2}) and the addresschecker reads both of them</p>

```

module de.codecentric.addresschecker{
     exports de.codecentric.addresschecker.api;
     requires de.codecentric.zipvalidator.v1;
     requires de.codecentric.zipvalidator.v2;
}

```

<p>Surely, this cannot compile either? Reading two modules that export the same packages? Actually, it does. I was surprised by this – the compiler acknowledges the situation, but settles for warnings like this one:</p><div>

```bash
./de.cc.zipvalidator.v1/de/codecentric/zipvalidator/api/ZipCodeValidator.java:1: 
warning: package exists in another module: de.codecentric.zipvalidator.v2
```

</div><p>Being developers, a warning is readily ignored and the application is run. Jigsaw really doesn’t like what it sees at runtime, though:</p><div>

```bash
java.lang.module.ResolutionException: 
Modules de.codecentric.zipvalidator.v2 and de.codecentric.zipvalidator.v1 export 
package de.codecentric.zipvalidator.api to module de.codecentric.addresschecker
```

</div><p>I personally think this is not intuitive and a compile time error might be nicer. I asked on the <a href="http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-November/005244.html">mailing list</a>  about the motivation behind this choice, but have yet to receive a response at the time of writing.</p><h3>Automatic modules and the unnamed module</h3><p>So far, we have been working in an all-module environment. But what about those very likely cases where we have to work with non-modular Jar files? This is where <strong>automatic modules</strong> and the <strong>unnamed module</strong> come into play.</p><p>Let’s start with <strong>automatic modules</strong>. An automatic module is a jar that you put on the modulepath. Once you put it there, it answers the three questions like this</p><p><strong>Q: What’s it’s name?</strong><br /> A: It is the name of the jar. So if you put guava.jar on the modulepath, you get an automatic module called guava. This also means that you cannot use a Jar straight from a Maven repository because guava-18.0 is not a valid Java identifier.</p><p><strong>Q: What does it export?</strong><br /> A: An automatic module exports all its packages. So all public types are available to every module that reads an automatic module.</p><p><strong>Q: What does it require?</strong><br /> A: An automatic module reads *all* other available modules (including the unnamed module, more on that later). This is important! You can access all exported types of any other module from an automatic module. You do not have to specifiy this anywhere, it is implied.</p><p>Let’s try an example. We start to use com.google.common.base.Strings in the zipvalidator. To allow this access, we must define a read edge to Guava’s automatic module:</p>

```

module de.codecentric.zipvalidator{
       exports de.codecentric.zipvalidator.api;
       requires public de.codecentric.zipvalidator.model;
       requires guava;

 }

```

<p>For compilation, we need to put the guava.jar on the modulepath (it’s in folder ../jars):</p><div>

```bash
javac -d . -modulepath ../jars -modulesourcepath .  $(find . -name "*.java")
```

</div><p>This compiles and runs just fine.</p><p>(For the record, this example was not easy to get working. I got into some issues with this using Jigsaw build 86 because it complained about dependencies to a module called jdk.management.resource. I asked on the mailing list about this, you can see the conversation <a href="http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-November/005190.html">here</a> .<br /> Basically, the solution was not using the early access build but building the JDK myself. There were some more issues with this on OSX Mavericks as you can see in the thread, I had to change a makefile, but in the end I got it working. Your mileage may vary with later releases).</p><p>This is a good time to introduce you to your new best friend during the transition to Jigsaw. There is a tool called jdeps that takes a look at your non-modular code and tells you about its dependencies. Let’s look at guava:</p><div>

```bash
jdeps -s ../jars/guava.jar
```

</div><p> has the following output</p><div>

```bash
guava.jar -> java.base
guava.jar -> java.logging
guava.jar -> not found
```

</div><p>This means that the automatic guava module requires java.base, java.logging and … “not found?! What is that? Leaving out the -s switch, jdeps leaves the module perspective and goes down a step to package level (shortened for brevity, guava has quite a few packages):</p><div>

```bash
   com.google.common.xml (guava.jar)
      -> com.google.common.escape                           guava.jar
      -> java.lang
      -> javax.annotation                                   not found
```

</div><p>Here, we see that the com.google.common.xml package depends on com.google.common.escape which it located in the module itself, java.lang which is well-known and javax.annotation which is not found. This tells us that we need a jar containing JSR-305 types as this contains javax.annotation (I actually do not do this for these examples – i don’t need any type from this package in my examples and neither the compiler nor the runtime cares).</p><h3>The unnamed module</h3><p>So what is the <strong>unnamed module</strong>? Let’s answer the three questions again:</p><p><strong>Q: What’s it’s name?</strong><br /> A: If you haven’t guessed yet, the unnamed module doesn’t have a name</p><p><strong>Q: What does it export?</strong><br /> A: The unnamed module exports all its packages to any other module. That does not mean you can read it from any other module – it doesn’t have a name, so you cannot require it! requires unnamed; does not work.</p><p><strong>Q: What does it require?</strong><br /> A: The unnamed module reads <strong>all</strong> other available modules.</p><p>So, if you cannot read the unnamed module from any of your modules, what’s the point? To answer this, we meet an old friend – the classpath. Every type read from the classpath (instead of the modulepath) is automatically placed in the unnamed module – or differently put, every type in the unnamed module has been loaded via classpath. Since the unnamed module reads every other module, we can access all exported types from any classpath-loaded type. Java 9 will support use of classpath and modulepath either in isolation or even mixed to ensure downward compatibility. Let’s look at some examples.</p><p>Let’s assume we still have our nice zipvalidator module, but our addresschecker is still non-modular and does not have a module-info.java. This is our source structure</p>

```

one-module-with-unnamed-ok/
├── classpath
│   └── de.codecentric.legacy.addresschecker
│       └── de
│           └── codecentric
│               └── legacy
│                   └── addresschecker
│                       ├── api
│                       │   ├── AddressChecker.java
│                       │   └── Run.java
│                       └── internal
│                           └── AddressCheckerImpl.java
├── modulepath
│   └── de.codecentric.zipvalidator
│       ├── de
│       │   └── codecentric
│       │       └── zipvalidator
│       │           ├── api
│       │           │   ├── ZipCodeValidator.java
│       │           │   └── ZipCodeValidatorFactory.java
│       │           └── internal
│       │               └── ZipCodeValidatorImpl.java
│       └── module-info.java


```

<p>There is now one folder called <strong>classpath</strong> that contains the legacy code that wants to access the zipvalidator, and a folder called <strong>modulepath</strong> that contains the zipvalidator module. We can compile our modules in the usual way. To compile the legacy code, we need to provide info of the modular code. We do this by putting it on the classpath:</p><div>

```bash
javac -d classpath/de.codecentric.legacy.addresschecker  
  -classpath modulepath/de.codecentric.zipvalidator/ $(find classpath -name "*.java")
```

</div><p>This just works as usual.</p><p>At runtime, we have two options now. We can</p><ul><li>put the module on the classpath</li><li>mix classpath and modulepath</li></ul><p>Using the first option effectively means that we do not use the module system. All types are put in the unnamed module where they can freely access each other.</p><div>

```bash
java -cp modulepath/de.cc.zipvalidator/:classpath/de.cc.legacy.addresschecker/
    de.codecentric.legacy.addresschecker.api.Run 76185
```

</div><p>behaves exactly like the java application you’re using today.</p><p>Mixing classpath and module path on the other hand works like this</p><div>

```bash
java -modulepath modulepath  -addmods de.codecentric.zipvalidator 
    -classpath classpath/de.codecentric.legacy.addresschecker/ 
    de.codecentric.legacy.addresschecker.api.Run
```

</div><p>We use both the -classpath and -modulepath switches. A new addition is the -addmods switch – when mixing classpath and modulepath, we don’t just get access to any module in the modulepath folders, we have to specifically state which ones should be available.</p><p>This approach works fine too, but there is a caveat! Remember, the answer to “what does the unnamed module require” is “all other modules”. If we use the zipvalidator module via the modulepath, we can only use its <strong>exported</strong> packages. Anything else will result in an IllegalAccessError at runtime. So you have to stick to the rules of the module system in this case.</p><h3>Creating runtime images with jlink</h3><p>That’s it for the module examples, but there is another new tool that deserves our attention. jlink is Java 9’s utility to create your own JVM distributions. The cool thing is that due to the newly modularized nature of the JDK, you can pick which modules you want to include in this distribution! Let’s see an example. If we want to create a runtime image that includes our addresschecker, we issue the command</p><div>

```bash
jlink --modulepath $JAVA9_BIN/../../images/jmods/:two-modules-ok/ 
    --addmods de.codecentric.addresschecker --output linkedjdk
```

</div><p>We only specify three things:</p><ul><li>the modulepath (including your custom modules and the path to the jmods folder in your JDK – this includes the standard java modules)</li><li>the modules you want to include in your distribution</li><li>an output folder</li></ul><p>This command creates the following:</p>

```

linkedjdk/
├── bin
│   ├── java
│   └── keytool
├── conf
│   ├── net.properties
│   └── security
│       ├── java.policy
│       └── java.security
└── lib
    ├── classlist
    ├── jli
    │   └── libjli.dylib
    ├── jspawnhelper
    ├── jvm.cfg
    ├── libjava.dylib
    ├── libjimage.dylib
    ├── libjsig.diz
    ├── libjsig.dylib
    ├── libnet.dylib
    ├── libnio.dylib
    ├── libosxsecurity.dylib
    ├── libverify.dylib
    ├── libzip.dylib
    ├── modules
    │   └── bootmodules.jimage
    ├── security
    │   ├── US_export_policy.jar
    │   ├── blacklisted.certs
    │   ├── cacerts
    │   └── local_policy.jar
    ├── server
    │   ├── Xusage.txt
    │   ├── libjsig.diz
    │   ├── libjsig.dylib
    │   ├── libjvm.diz
    │   └── libjvm.dylib
    └── tzdb.dat

```

<p>That’s it. On OSX Mavericks, this is about 47 MB in size. We can also enable compression and remove some debugging features that you wouldn’t need on a production system anyway. The smallest distribution I have managed so far was using the following command:</p><div>

```bash
jlink --modulepath $JAVA9_BIN/../../images/jmods/:two-modules-ok/bin 
    --addmods de.codecentric.addresschecker --output linkedjdk --exclude-files *.diz 
    --compress-resources on --strip-java-debug on --compress-resources-level 2
```

</div><p>This reduces the distribution to about 18 MB, which sounds pretty cool to me. Apparently you can get it down to 13 on Linux</p><p>Calling</p><div>

```bash
/bin/java --listmods
```

</div><p> shows the modules that are contained in this distribution</p><div>

```bash
de.codecentric.addresschecker
de.codecentric.zipvalidator
java.base@9.0
```

</div><p>So all applications that depend on a maximum of those modules can run on this JVM. I have not been able to get our main class to run in this scenario though. To do this, I used another way:</p><p>A keen observer might have noticed that the second call to jlink had a different modulepath than the first one. In the second one, we specify a path to a folder named <strong>bin</strong>. This folder contains modular jars, and jar for the addresschecker also contains a main-class info in its Manifest. jlink uses this information to add an extra entry to our JVM bin folder:</p>

```

linkedjdk/
├── bin
│   ├── de.codecentric.addresschecker
│   ├── java
│   └── keytool
...

```

<p>This means that now, we can call our application directly. Sweet!</p><div>

```bash
./linkedjdk/bin/de.codecentric.addresschecker 76185
```

</div><p> prints</p><div>

```bash
76185 is a valid zip code
```

</div><h3>Conclusion</h3><p>This concludes our introduction to Jigsaw. We went through some examples showing what you can and cannot do with Jigsaw and Java 9. Jigsaw will be a somewhat disruptive change that cannot be simply picked up like Lambdas or try-with-resources. Our whole tool chain from build tools like Maven or Gradle to IDEs will need to adapt to the module system. At JavaOne, Hans Dockter of Gradle Inc. held a session showing how you can already start writing modular code even with Java <9 -="" Gradle="" will="" perform="" checks="" at="" compile="" time="" and="" fail="" if="" module="" integrity="" is="" violated.="" This="" (experimental)="" feature="" has="" been="" included="" in="" the="" recent="" release="" of="" Gradle 2.9. Interesting times are certainly ahead!</p><p>For more information about Jigsaw I once more recommend <a href="http://openjdk.java.net/projects/jigsaw/" target="_blank">the Jigsaw Project home page</a> , especially the <a href="http://openjdk.java.net/projects/jigsaw/" target="_blank">slides and videos</a>  from this year’s JavaOne sessions on Jigsaw and project lead Mark Reinhold’s http://openjdk.java.net/projects/jigsaw/spec/sotms/.</p>

First steps with Java 9 and Project Jigsaw – Part 2

Realtime Fast Data Analytics with Druid

<p>Messages in Apache Kafka are appended to (partitions of) a topic. Topics have a partition count, a replication factor and various other configuration values. Why do those matter and what could possibly go wrong?</p><h2>Why does Kafka topic configuration matter?</h2><p>There are three main parts that define the configuration of a Kafka topic:</p><ul><li>Partition count</li><li>Replication factor</li><li>Technical configuration</li></ul><p>The partition count defines the level of parallelism of the topic. For example, a partition count of 50 means that up to 50 consumer instances in a consumer group can process messages in parallel. The replication factor specifes how many copies of a partition are held in the cluster to enable failover in case of broker failure. And in the technical configuration, one can define the cleanup policy (deletion or log compaction), flushing of data to disk, maximum message size, permitting unclean leader elections and so on. For a complete list, see <a href="https://kafka.apache.org/documentation/#topicconfigs" target="_blank">https://kafka.apache.org/documentation/#topicconfigs</a> . Some of these properties are quite easy to change at runtime. For others this is a lot harder, though.</p><p>Let’s take the partition count. Increasing it upwards is easy – just run</p>

```
bin/kafka-topics.sh --alter --zookeeper zk:2181 --topic mytopic --partitions 42
```

<p> This might be sufficient for you. Or it might open the fiery gates of hell and break your application. The latter is the case if you depend on all messages for a given key landing on the same partition (to be handled by the same consumer in a group) or for example if you run a Kafka Streams application. If that application <a href="/wissens-hub/blog/crossing-streams-joins-apache-kafka" target="_blank">uses joins</a> , the involved topics need to be copartitioned, meaning that they need to have the same partition count (and producers using the same partitioner, but that is hard to enforce). Even without joins, you don’t want messages with the same key end up in different <a href="https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KTable.html" target="_blank">KTables</a> .</p><p>Changing the replication factor is serious business. It is not a case of simply saying “please increase the replication factor to x” as it is with the partition count. You need to completely reassign partitions to brokers, specifying the preferred leader and n replicas for each partition. It is your task to distribute those well across your cluster. This is no fun for anyone involved. Practical experience with this has actually led to this blog post.<br /> The technical configuration has an impact as well. It could be for example quite essential that a topic is using compaction instead of deletion if an application depends on that. You also might find the retention time too small or too big.</p><h2>The Evils of Automatic Topic Creation</h2><p>In a recent project, a central team managed the Kafka cluster. This team kept a lot of default values in the broker configuration. This is mostly sensible as Kafka comes with pretty good defaults. However, one thing they kept was <code>auto.create.topics.enable=true</code>. This property means that whenever a client tries to write to or read from a non-existing topic, Kafka will automatically create it. Defaults for partition count and replication factor were kept at 1.</p><p>This led to the situation where the team forgot to set up a new topic manually before running producers and consumers. Kafka created that topic with default configuration. Once this was noticed, all applications were stopped and the topic deleted – only to be created again automatically seconds later, presumably because the team didn’t find all clients. “Ok”, they thought, “let’s fix it manually”. They increased the partition count to 32, only to realize that they had to provide the complete partition assignment map to fix the replication factor. Even with tool support from Kafka Manager, this didn’t give the team members a great feeling. Luckily, this was only a development cluster, so nothing really bad happened. But it was easy to conceive that this could also happen in production as there are no safeguards.</p><p>Another danger of automatic topic creation is the sensitivity to typos. Let’s face it – sometimes we all suffer from butterfingers. Even if you took all necessary care to correctly create a topic called “parameters”, you might end up with something like</p><p></p><p>Automatic topic creating means that your producer thinks everything is fine, and you’ll scratch your head as to why your consumers don’t receive any data.</p><p>Another conceivable issue is that a developer that maybe is not yet that familiar with the Producer API might confuse the String parameters in the <em>send</em> method</p><p></p><p>So while our developer meant to assign a random value to the message key, he accidentally set a random topic name. Every time a message is produced, Kafka creates a new topic.</p><p>So why don’t we just switch automatic topic creation off? Well, if you can: do it. <strong>Do it now!</strong> Sadly, the team didn’t have that option. But an idea was born – what would be the easiest way to at least fail fast at application startup when something is different than expected?</p><h2>How to automatically check your topic configuration</h2><p>In older versions of Kafka, we basically used the code called by the <code>kafka-topics.sh</code> script to programmatically work with topics. To create a topic for example we looked at how to use <code>kafka.admin.CreateTopicCommand</code>. This was definitely better than writing straight to Zookeeper because there is no need to replicate the logic of “which ZNode goes where”, but it always felt like a hack. And of course we got a dependency on the Kafka broker in our code – definitely not great.</p><p>Kafka 0.11 implemented <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations" target="_blank">KIP-117</a> , thus providing a new type of Kafka client – <a href="https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html" target="_blank"><code>org.apache.kafka.clients.admin.AdminClient</code></a> . This client enables users to programmatically execute admin tasks without relying on those old internal classes or even Zookeeper – all Zookeeper tasks are executed by brokers.</p><p>With AdminClient, it’s fairly easy to query the cluster for the current configuration of a topic. For example, this is the code to find out if a topic exists and what its partition count and replication factor is:</p><p></p><p>The <a href="https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/admin/DescribeTopicsResult.html" target="_blank"><code>DescribeTopicsResult</code></a>  contains all the info required to find out if the topic exists and how partition count and replication factor are set. It’s asynchronous, so be prepared to work with Futures to get your info.</p><p>Getting configs like <code>cleanup.policy</code> works similarly, but uses a different method:</p><p></p><p>Under the hood there is the same Future-based mechanism.</p><h2>A first implementation attempt</h2><p>If you are in a situation where your application depends on a certain configuration for the Kafka topics you use, it might make sense to fail early when something is not right. You get instant feedback and have a chance to fix the problem. Or you might at least want to emit a warning in your log. In any case, as nice as the AdminClient is, this check is not something you should have to implement yourself in every project.</p><p>Thus, the idea for a small library was born. And since naming things is hard, it’s called “Club Topicana”.</p><p>With Club Topicana, you can check your topic configuration every time you create a Kafka Producer, Consumer or Streams client.</p><p>Expectations can be expressed programmatically or configuratively. Programmatically, it uses a builder:</p><p></p><p>This basically says “I expect the topic test_topic to exist. It should also have 32 partitions and a replication factor of 3. I also expect the cleanup policy to be delete. Kafka should retain messages for at least 30 seconds.”</p><p>Another option to specify an expected configuration is YAML (parser is included):</p><p></p><p>What do you do with those expectations? The library provides factories for all Kafka clients that mirror their public constructors and additionally expects a collection of expected topic configurations. For example, creating a producer can look like this:</p><p></p><p>The last line throws a MismatchedTopicConfigException if the actual configuration does not meet expectations. The message of that exception lists the differences. It also provides access to the computed result so users can react to it in any way they want.</p><p>The code for consumers and streams clients looks similar. Examples are available on <a href="https://github.com/ftrossbach/club-topicana" target="_blank">GitHub</a> . If all standard clients are created using Club Topicana, an exception will prevent creation of a client and thus auto creation of a topic. Even if auto creation is disabled, it might be valuable to ensure that topics have the correct configuration.</p><p>There is also a Spring client. The <code>@EnableClubTopicana</code> annotation triggers Club Topicana to read YAML configuration and execute the checks. You can configure if you want to just log any mismatches or if you want to let the creation of the application context fail.</p><p>This is all on <a href="https://github.com/ftrossbach/club-topicana" target="_blank">GitHub</a>  and available on Maven Central.</p><h2>Caveats</h2><p>Club Topicana will not notice when someone changes the configuration of a topic after your application has successfully started. It also of course cannot guard against other clients doing whatever on Kafka.</p><h2>Summary</h2><p>The configuration of your Kafka topics is an essential part of running your Kafka applications. Wrong partition count? You might not get the parallelism you need or your streams application might not even start. Wrong replication factor? Data loss is a real possibility. Wrong cleanup policy? You might lose messages that you depend on later. Sometimes, your topics might be auto-generated and come with bad defaults that you have to fix manually. With the AdminClient introduced in Kafka 0.11, it’s simple to write a library that compares actual and desired topic configurations at application startup.</p>

Validating Topic Configurations in Apache Kafka

Living on the edge: building serverless applications with Cloudflare Workers

<p>Neuronale Netze sind für Außenstehende häufig von einer mystischen Aura umgeben. Sie werden regelmäßig in Verbindung mit menschlichen Gehirnen gebracht, und ihnen wird eine sich verselbständigende Intelligenz zugeschrieben. Das macht sie für viele mysteriös und suspekt. Um ein wenig zur Entmystifizierung beizutragen, widmen wir uns in diesem Blogpost den grundlegenden Prinzipien, auf denen der Trainingsprozess eines neuronalen Netzes beruht. Der Post richtet sich insbesondere an interessierte Personen mit wenig Vorwissen, und wir werden daher auf alle mathematische Details verzichten.</p><p>Im Folgenden präzisieren wir zunächst die Ausgangssituation. Dann lernen wir die Feedback-Schleife kennen, welche das Herzstück des Trainings darstellt. Im Anschluss betrachten wir die grundsätzlichen Fragen, die ein Algorithmus zum Trainieren eines neuronales Netzes beantworten sollte. Abschließend beschäftigen wir uns damit, wie das Training mit einer großen Anzahl an Daten funktionieren kann und wann das Training eines neuronalen Netzes eigentlich abgeschlossen ist.</p><h2>Die Ausgangssituation</h2><p>Um die Komplexität überschaubar zu halten, betrachten wir das neuronale Netz schlicht als eine Black Box mit einer Anzahl verschiedener Regler. Unsere Black Box „Neuronales Netz“ (BB-NN) verfügt über die Fähigkeit, Aussagen über einen bestimmten Sachverhalt zu treffen. In unserer Vorstellung können wir an den Reglern drehen und dadurch die Aussagen der BB-NN beeinflussen. In der Sprache der neuronalen Netze entsprechen die Regler der BB-NN den <em>Gewichten</em> des Netzwerks</p><p>Konkret stellen wir uns dabei folgende Situation vor: Wir zeigen unserer BB-NN ein Bild von einer Person und erhalten von ihr eine Aussage darüber, ob die abgebildete Person eine Brille trägt oder nicht. Hierbei ist wichtig festzuhalten, dass diese Aussage natürlich richtig oder falsch sein kann. Zu Beginn weiß unsere BB-NN noch nichts darüber, was wir eigentlich von ihr haben wollen, und sie wird mit hoher Wahrscheinlichkeit jede Menge falsche Aussagen treffen.</p><p>Unter Trainieren verstehen wir in unserem Kontext nun, die Regler unserer BB-NN so einzustellen, dass sie möglichst viele richtige Aussagen macht. Also möglichst häufig erkennt, ob die Person im Bild eine Brille trägt oder nicht. Wir gehen im Folgenden davon aus, dass wir eine große Anzahl an geeigneten Bildern besitzen, die wir unserer BB-NN zeigen und dabei anschließend überprüfen können, ob die von ihr getroffene Aussage — „trägt Brille“ oder „trägt keine Brille“ — korrekt ist. In der Praxis bedeutet dies, dass wir zum Trainieren eines neuronalen Netzes üblicherweise eine große Menge gelabelter Daten benötigen.</p><h2>Die Feedback-Schleife</h2><p>Der Hauptmechanismus zum Trainieren unserer BB-NN kann durch die Feedback-Schleife</p><p><img width="250" height="102" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/KTP1OfUiS6e1aQ8k4Axf" alt="Grafik Feedback-Schleife" /></p><p>visualisiert werden. Die grundlegende Idee hierbei ist, dass wir unserer BB-NN die Bilder zeigen und uns deren Aussagen zu den einzelnen Bildern notieren. Anschließend überprüfen wir die Korrektheit aller so erhaltener Aussagen. Im nächsten Schritt drehen wir an den Reglern unserer BB-NN mit dem Ziel, eine größere Anzahl korrekter Aussagen zu erhalten. Danach wiederholen wir diesen Vorgang. Dabei hoffen wir, dass wir die Regler nach und nach so einstellen können, dass wir mehr und mehr korrekte Aussagen von unserer BB-NN erhalten. In der Sprache der neuronalen Netze wird ein einzelner Durchlauf durch die Feedback-Schleife häufig als eine <em>Epoche</em> bezeichnet. Je größer also die Anzahl der Epochen eines neuronalen Netzes ist, desto länger wurde es trainiert.</p><h2>Trainingsalgorithmen</h2><p>Natürlich ist man in der Praxis nicht selbst damit beschäftigt, die Feinjustierung der verschiedenen Regler eines neuronales Netzes von Hand durchzuführen. Hierfür gibt es eine Vielzahl von Algorithmen, welche diese Aufgabe für uns übernehmen und zum Trainieren neuronaler Netze eingesetzt werden. Dabei bildet die Feedback-Schleife typischerweise das Fundament dieser Algorithmen. Die entscheidende Frage an dieser Stelle ist natürlich: Wie sollen die Regler eigentlich neu eingestellt werden? Oder allgemeiner: Wie soll das Update durchgeführt werden?</p><p>Aus der Vogelperspektive betrachtet, sollte ein Algorithmus zum Trainieren unserer BB-NN also mindestens eine Aussage über zwei Dinge liefern:</p><ul><li>In welche Richtung müssen die Regler gedreht werden?</li><li>Wie weit müssen die Regler gedreht werden?</li></ul><p>Die erschöpfende Antwort auf die erste Frage findet sich in der Mathematik — genauer in der Lösungstheorie von Optimierungsproblemen. An dieser Stelle halten wir aber einfach nur fest, dass es grundsätzlich möglich ist, eine mathematisch fundierte Aussage darüber zu treffen, in welche Richtung die Regler gedreht werden sollten.</p><p>(Interessierte Leser/innen sind eingeladen, sich als Einstieg in die zugrundeliegende Mathematik mit dem <a href="https://de.wikipedia.org/wiki/Gradientenverfahren" target="_blank">Gradientenverfahren</a>  auseinander zu setzen.)</p><p>Damit bleibt die Frage, wie weit wir die Regler drehen sollen. In der Sprache der neuronalen Netze entspricht “wie weit drehen” der sogenannten <em>Learning Rate</em>. Generell ist es dabei schwierig von DER Learning Rate zu sprechen, da bei der Auswahl der Learning Rate unter anderem auch folgende Fragen relevant sind:</p><ul><li>Sollte man am Anfang einen fixen Wert wählen; also die Regler in jeder Epoche gleich weit drehen, oder ist es besser, die Rate im Laufe des Trainings anzupassen?</li><li>Sollte man einen Wert festlegen, der für alle Regler gilt, oder ist es sinnvoller, verschiedene Werte für unterschiedliche Regler zu verwenden?</li></ul><p>Alle diese Fragen sind Gegenstand aktueller Forschung in diesem Fachbereich und können daher nicht pauschal beantwortet werden. Verschiedene Algorithmen verwenden hier unterschiedliche Strategien. Dabei ist ein zentrales Ziel aller Algorithmen, stets die Geschwindigkeit des Lernvorgangs zu erhöhen.</p><h2>Große Datenmengen</h2><p>Die Feedback-Schleife liefert uns eine erste gute Vorstellung davon, wie das Training eines neuronales Netzes funktioniert. In der Praxis werden neuronale Netze normalerweise mit einer sehr großen Anzahl an Daten trainiert. Dies hat zur Folge, dass ein Durchlauf durch die Feedback-Schleife mit dem gesamten Datensatz sehr teuer, d.h. sehr rechenaufwändig werden kann. Dies ist ein Grund, aus dem viele Algorithmen bei jedem einzelnen Durchlauf durch die Feedback-Schleife lediglich einen Teil der Daten, eine sogenannte <em>Batch</em>, verwenden. Dabei wird die Batch in jeder Epoche neu per Zufall aus den Daten zusammengestellt.</p><p>Die Feinjustierung des neuronalen Netzes basiert also in jeder Epoche lediglich auf einem Teil der Daten. Jedoch wird das neuronale Netz im Laufe des Trainings nach und nach immer wieder mit den gesamten Daten konfrontiert. Algorithmen, die diesen Mechanismus verwenden, erkennt man typischerweise an dem Zusatz <em>stochastic</em>.</p><h2>Genug trainiert</h2><p>An dieser Stelle lehnen wir uns kurz zurück, um darüber nachzudenken, was das Ziel des Trainings unserer BB-NN eigentlich ist: Wir wollen von unserer BB-NN am Ende eine möglichst korrekte Aussage über unbekannte Bilder bekommen. Also solche Bilder, die im Lernprozess keine Rolle gespielt haben. Überspitzt formuliert: Es ist uns ziemlich egal, wie genau unsere BB-NN die Bilder während des Trainings zuordnen kann. Wir wollen wissen, wie gut sie neue Bilder einordnen kann.</p><p>In einem Extremfall könnte unsere BB-NN während des Trainings einfach alle Bilder auswendig gelernt haben — und damit immer perfekt zuordnen. Eine Konsequenz aus der Spezialisierung eines neuronalen Netzes auf die Daten aus dem Training ist häufig, dass es anschließend Schwierigkeiten hat, mit unbekannten Daten umzugehen. Die Gefahr des Auswendiglernens während des Trainings wird in der Sprache der neuronalen Netze als <em>Overfitting</em> bezeichnet. Es stellt sich also die Frage, ob und wie man während des Trainings beobachten kann, ob das neuronale Netz einfach auswendig lernt.</p><h2>Aufteilung des Datensatzes</h2><p>Typischerweise wird aus diesem Grund ein Teil der vorhandenen Daten vor dem Lernprozess zur Seite gelegt und nicht beim Training berücksichtigt. Man spricht hierbei vom <em>training dataset</em> und <em>validation dataset</em>. Die Feedback-Schleife wird nun lediglich mithilfe der Daten des training dataset durchgeführt. Nach wie vor überprüfen wir dabei, wie gut das neuronale Netz diese vorhersagen kann. Verbessern sich diese Vorhersagen im Verlauf des Trainings, ist dies zunächst ein guter Indikator dafür, dass das Training funktioniert. Zusätzlich zum Abschneiden auf dem training dataset überprüfen wir aber außerdem noch, wie gut das neuronale Netz das validation dataset vorhersagen kann.</p><p>In unserem Beispiel würden wir demnach vor dem Training einen Teil der Bilder zur Seite legen. Beim Durchlaufen der Feedback-Schleife würden wir unsere BB-NN auch dazu nutzen, Aussagen über die beiseite gelegten Bilder zu treffen. Diese Bilder dürfen keinen direkten Einfluss darauf haben, wie die Regler in der Update-Phase neu justiert werden. Die Feinjustierung des neuronalen Netzes muss ausschließlich auf Basis der Trainingsdaten stattfinden.</p><p>Als Daumenregel kann man nun formulieren: Beende das Training, wenn das neuronale Netz beginnt, bessere Vorhersagen über das training dataset zu machen als über das validation dataset. Hinter diesem Abbruchkriterium steckt die Idee, das Training genau dann zu beenden, wenn das neuronale Netz anfängt, die Daten des training datasets auswendig zu lernen und dadurch schlicht zu verhindern, dass es sich auf das training dataset spezialisiert.</p><p>In der Praxis ist es üblich, neben dem validation dataset noch ein <em>test dataset</em> zu verwenden. Auch dieser Datensatz wird vor Beginn des Trainings abgespalten und zur Seite gelegt. Die Daten des test datasets werden nach Abschluss des Trainings lediglich einmal dazu benutzt, herauszufinden wie gut das fertig trainierte neuronale Netz auf ungesehenen Daten funktioniert. Für diese Daten gilt: Sie dürfen keinerlei Einfluss auf den gesamten Trainingsprozess des Netzes haben. Insbesondere dürfen sie auch nicht für das Abbruchkriterium des Trainings verwendet werden.</p><h2>Fazit</h2><p>In diesem Blogpost haben wir uns mit der Frage auseinandergesetzt, wie man ein neuronales Netz trainiert. Dazu haben wir die Feedback-Schleife kennengelernt, welche typischerweise dem Training zugrunde liegt. Wir haben uns Gedanken darüber gemacht, welche Antworten ein Algorithmus zum Training eines neuronalen Netzes liefern muss. Abschließend haben wir uns dem Thema Overfitting gewidmet und ein Abbruchkriterium für die Feedback-Schleife und damit des Trainings kennengelernt.</p><p>Ich hoffe, euch hat der kurze Ausflug in die Welt der neuronalen Netze gefallen. Wenn ihr mehr zum Thema künstliche Intelligenz erfahren möchtet, einfach <a href="https://www.codecentric.de/kuenstliche-intelligenz/">hier</a>  klicken.</p><p>Ihr habt Fragen zum Blogpost? Dann schreibt mir doch einfach einen Kommentar oder eine Mail.</p>

Wie trainiert man eigentlich neuronale Netze?

<h2>Einleitung</h2><p>Künstliche Intelligenz (KI) findet sich heutzutage scheinbar überall. Bereits ohne den derzeitigen Hype-Faktor um KI ist der Begriff nur schwer zu greifen. Viele Unternehmen sehen sich unter Zugzwang, KI als neue Technologie einzusetzen und deren Potentiale auszuschöpfen. Große Internet(!)-Konzerne haben das Geld, die Daten und Expertise in Data Science und zeigen, welche innovativen Produkte damit heute möglich sind.</p><p>Schon lange ist es zuverlässig möglich, Spam-E-Mails automatisch zu erkennen. Aktuelle KI-Technologien erlauben es Maschinen heute, E-Mails auch inhaltlich zu verstehen. Dadurch ergeben sich neue Möglichkeiten E-Mails automatisiert zu verarbeiten.</p><p>Darum geht es in diesem Artikel.</p><h2>Die Situation</h2><p>Stellen wir uns folgende Situation vor: die ACME Mobilfunkgesellschaft erhält täglich mehrere Tausend Kunden-E-Mails mit unterschiedlichsten Anliegen. Beim Großteil der Anliegen (~80%) handelt es sich um einfache Sachverhalte wie „Stammdatenänderung“, „Vertragsänderungen“ oder „Kündigungen“. Die restlichen ~20% der E-Mails sind komplexere Anliegen, etwa individuelle offene Fragen.</p><p>Um diese E-Mails zu verarbeiten und zu beantworten hat ACME eine eigene Customer-Support-Abteilung mit 952 Mitarbeiten. Je weiter ACME es schafft, die einfachen Sachverhalte automatisiert zu bearbeiten, desto mehr Zeit haben die Mitarbeiter, um individuelle Kundenanfragen personalisiert und kompetent zu beantworten.</p><p>Die Support-Abteilung hat ein typisches CRM-System, welches die Abteilung unterstützt. Es ist regelbasiert, d.h. E-Mails wurden lediglich nach bestimmten Schlagworten wie „Kündigung“ oder „Vertragsänderung“ durchsucht, um deren Intention zu erkennen. Das war bis vor kurzem kaum anders möglich.</p><p>Heute erlaubt es jedoch moderne NLP-Technologie, Systeme zu entwickeln, die den Kontext von E-Mails begreifen können. Diesen Fortschritt könnte ACME nutzen, um E-Mails verlässlicher einzuordnen und zu verarbeiten und Schritt für Schritt auch komplexere Sachverhalte zu automatisieren. Es ist sogar möglich geworden, mehrere Intentionen aus einer Freitext-E-Mail zuverlässig zu erkennen.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/BRLAfgR6TwOURO4wJWwK"><img width="250" height="90" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/BRLAfgR6TwOURO4wJWwK" alt="Textklassifizierung" /></a> </p><h2>Das KI-System</h2><p>Im Customer-Support bietet sich für Unternehmen häufig eine gute Möglichkeit, iterativ ein KI-System einzuführen und weiterzuentwickeln.</p><p>Das Ziel ist es hierbei mithilfe des KI-Systems, die automatisierte Verarbeitung von E-Mails zu verbessern. In der ersten Ausbaustufe könnte das System zunächst in der Lage sein, das Anliegen von Kunden-E-Mails zu erkennen und diese automatisiert an die richtige Stelle im Unternehmen weiterzuleiten.</p><p>Sobald dieses Basissystem produktiv ist, könnte dann begonnen werden, es inkrementell zu erweitern. Denkbar wären hier z.B. folgende Features:</p><ul><li>vorausgefüllte E-Mail Templates für Support-Mitarbeiter zur Beantwortung der E-Mails</li><li>sofortige, personalisierte und individuelle Rückmeldung des Systems an den Kunden</li><li>automatisierte Beantwortung einfacher Anfragen</li><li>automatisierte Bearbeitung einfacher Aufgaben, z.B. Update der Kundendaten bei Adressänderung</li></ul><p>Ein zusätzlicher Mehrwert kann darüber hinaus oft durch automatisierte Analysen der Kundenanfragen generiert werden.</p><ul><li>Welche Probleme hat die Kundschaft an einem bestimmten Tag beschäftigt?</li><li>Welche Probleme können frühzeitig erkannt und behoben werden?</li></ul><p>Unsere Projekterfahrung hat gezeigt, dass KI-unterstützte Support-Arbeitsplätze zu einer drastischer Verkürzung der Bearbeitungszeit von Kundenanfragen führen können. Durch Minimierung repetitiver Aufgaben kann zudem die Mitarbeiterzufriedenheit deutlich gesteigert werden.</p><h2>Löst KI jetzt alle Probleme?</h2><p>Häufig herrscht leider eine falsche Erwartungshaltung. KI wird mit menschlicher Intelligenz verglichen – davon sind wir aber noch weit entfernt. KI kann derzeit repetitive Aufgaben erlernen, aber noch lange nicht denken wie ein Mensch.</p><p>In einem ersten Impuls besteht die initiale Idee für ein KI-System oft darin, sehr komplexe Prozesse im Herzstück eines Unternehmens zu verbessern. Dies ist meist dem Hintergedanken geschuldet, sofort den maximalen Nutzen aus der neuen Technologie zu ziehen. Häufig befinden sich an diesen Stellen des Unternehmens auch tatsächlich große Hebel mit enormen Potenzialen, die perspektivisch genutzt werden können. Jedoch ist die Vorstellung, solche Prozesse durch ein KI-System zu verbessern, in vielen Fällen unrealistisch.</p><p>Dies liegt daran, dass die meisten relevanten Unternehmensprozesse meist schon über Jahre hinweg ohne KI, aber mit enormem Know-How-Einsatz sehr stark optimiert wurden. Zusätzlich fehlt oft jegliche Erfahrung mit KI-Systemen und die vorhandene IT-Infrastruktur ist nicht darauf vorbereitet, komplexe KI-Systeme zu entwickeln. Es kann daher fraglich sein, ob unter diesen Rahmenbedingungen ein erfolgreicher Schritt in Richtung KI möglich ist.</p><p>Eine Alternative hierzu ist es, zunächst an einem einfachen End-to-End KI-System Erfahrung über deren Entwicklung und Lebenszyklus zu sammeln. Neben dem gewünschten Erfahrungsaufbau zu KI-Systemen und einer Weiterentwicklung der IT-Infrastruktur, kann auch so bereits zusätzlicher Business Value mit KI generiert werden.</p><h2>Vorgehensweise</h2><p>Es ist ratsam, den abgegrenzten Use Case zunächst ohne zu viele Abhängigkeiten zu starten. Oft beobachten wir, dass erste Ziele zu hoch gesteckt werden und man sich in der Komplexität der Aufgabe verliert. Daher sollte der Fokus ausdrücklich darauf liegen, ein KI-System produktiv zu nehmen und dessen Lebenszyklus zu verstehen.</p><p>Für unser vorheriges Beispiel im Customer Support könnte dies bedeuten, dass man sich zunächst z.B. auf das Erkennen von Stammdatenänderung, Vertragsverlängerung und offenen Fragen konzentriert und die dafür nötige Infrastruktur aufbaut.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/jR558MWIRDi73rXf2Y7h"><img loading="lazy" width="250" height="75" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/jR558MWIRDi73rXf2Y7h" alt=" Künstliche Intelligenz" /></a> </p><p>Grundlage für die Entwicklung des Basissystems sind die folgenden vier Kernpunkte:</p><ul><li>Daten beschaffen und labeln</li><li>KI-System integrieren</li><li>Das Machine-Learning-Modell</li><li>Human assisted Feedback Loop</li></ul><p>Diese Kernpunkte werden wir nun etwas genauer beleuchten.</p><p><strong>Daten beschaffen und labeln</strong><br /> Ein entscheidender Schritt für KI-Systeme ist die Verfügbarkeit einer geeigneten Datenbasis. Um ein KI-System zur E-Mail Klassifikation zu trainieren, werden gelabelte Kunden-E-Mails benötigt. Das sind E-Mails, die bereits in eine Kategorie eingeordnet sind. Hierbei kann es in der Praxis eine erste Herausforderung sein, geeignete Kategorien festzulegen. Während z.B. „Stammdatenänderung“ noch recht generisch sind, spielt darüber hinaus schnell der konkrete Unternehmenskontext eine große Rolle.</p><p>Um das KI-System zu trainieren, wird insbesondere eine ausreichende Anzahl von E-Mails je Kategorie benötigt, um verlässliche Resultate erhalten zu können. Deswegen kann es sinnvoll sein, zunächst mit einer kleineren Menge an Kategorien zu starten, um diese dann später um weitere Kategorien zu ergänzen.</p><p><strong>KI-System integrieren</strong><br /> Ein häufig unterschätzter Fakt ist, dass KI-Systeme letztendlich Softwareprojekte sind. Um tatsächlich Business Value mithilfe eines KI-Systems zu generieren, muss es in Produktion gebracht werden. KI-Systeme enthalten zwar eine Machine-Learning-Komponente, diese ist allerdings nur ein Teil des Gesamtsystems. Bei allem Hype um all die neuen Möglichkeiten, die Neuronale Netze und moderne Technologien bieten, sollte daher das Gesamtbild nicht aus dem Blick verloren werden.</p><p>Auf den Use Case bezogen bedeutet dies, dass das Machine-Learning-Modell auf eine geeignete Weise deployed werden muss. Darüber hinaus müssen die eingehenden Kunden-E-Mails zum Modell gebracht werden, um diese dort zu klassifizieren. Anschließend sollten die klassifizierten E-Mails – mit dem entsprechenden Label versehen – an die richtige Stelle zur Weiterverarbeitung weitergeleitet werden. Die Gesamtheit all dieser Komponenten ist das eigentliche KI-System, welches dann auch in der Lage ist, einen Mehrwert zu generieren.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Yxu8EVRYRlaepRgX56bw"><img loading="lazy" width="250" height="93" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Yxu8EVRYRlaepRgX56bw" alt="Künstliche Intelligenz" /></a> </p><p><strong>Das Machine-Learning-Modell</strong><br /> Für viele Use Cases ist die Aufgabe ein geeignetes Machine-Learning-Modell zu finden äußerst komplex. Je nach Anforderung kann hier auch ein Deep-Learning-Modell – ein Modell basierend auf Neuronalen Netzen – erforderlich sein. Um ein solches Modell selbst zu entwickeln, wird ausreichend Data Science Know-How benötigt. Typischerweise gibt es hierfür eine Explorationsphase von ein paar Wochen, in denen Modelle evaluiert und schließlich ein erster Modell-Prototyp entwickelt wird.</p><p>Glücklicherweise ist E-Mail-Klassifikation schlicht Textklassifikation und das Problem ist im Deep-Learning-Bereich schon sehr gut verstanden. Daher besteht ein Vorteil unseres Beispiel-Use-Cases darin, dass es bereits fertige Modelle gibt, die man direkt verwenden kann. Es ist also möglich, für das vorgeschlagene Basis-KI-System ein bestehendes Open-Source-Modell zu verwenden, das von Data-Science-Experten unter Beachtung aktueller Best Practises entwickelt wurde.</p><p>Ein fertiges Modell kann beispielsweise aus dem beliebten Python-Modul spaCy entnommen und direkt trainiert werden.</p><p><strong>Human assisted Feedback Loop</strong><br /> Auch KI-Systeme machen Fehler. Sobald das KI-System produktiv ist, werden einige der eintreffenden E-Mails falsch klassifiziert werden. Das ist völlig normal und wird sich erfahrungsgemäß auch nicht vollständig vermeiden lassen. Auch ein Mensch wird in der Praxis Schwierigkeiten haben, alle eintreffenden Kunden-E-Mails stets korrekt einzuordnen. Und eine Gruppe von Personen wird bei einzelnen E-Mails unterschiedlicher Meinung sein, welche Klassifizierung richtig ist.</p><p>Ein KI-System kann sein volles Potenzial dann entfalten, wenn es sich regelmäßig weiter verbessern kann. Es ist sinnvoll, die klassifizierten E-Mails des Systems von einem Menschen verifizieren zu lassen. Dabei ist es wichtig, falsch zugeordnete E-Mails nicht einfach verärgert hinzunehmen, sondern diese mit dem richtigen Label zu versehen. Solche händisch korrigierten E-Mails sind sehr wertvoll und können gemeinsam mit den restlichen klassifizierten E-Mails dazu verwendet werden, die Anzahl an gelabelten E-Mails immer weiter zu vergrößern.</p><p>Dadurch vergrößert sich die Datengrundlage für das Training der Machine-Learning-Komponente ständig. Ab einer gewissen Menge an zusätzlichen Daten lohnt es sich, die Komponente erneut zu trainieren, um diese zu verbessern. Idealerweise ist das KI-System so konstruiert, dass es sich selbst regelmäßig neu trainiert und deployed.</p><p>So kann das KI-System immer besser und die Anzahl an händischen Korrekturen immer weiter reduziert werden.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HoI6c90QeOTokeERUM7h"><img loading="lazy" width="250" height="148" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/HoI6c90QeOTokeERUM7h" alt="Künstliche Intelligenz" /></a> </p><h2>Fazit</h2><p>Es ist sinnvoll, zunächst ein einfaches KI-System vollständig umzusetzen und dieses inkrementell weiter zu entwickeln. KI-Systeme sind Softwareprojekte, welche eine Machine-Learning-Komponente beinhalten. Um einen wirklichen Mehrwert zu erzeugen benötigt man neben Data Science Know-How auch Wissen über Daten-Pipelines und Continuous-Deployment-Strategien. Es ist ratsam, anhand eines einfachen KI-Systems und out-of-the-box Modells Erfahrung über den Lebenszyklus von KI-Systemen zu sammeln. Mit den daraus gewonnenen Erkenntnissen fällt es dann deutlich leichter, komplexere KI-Systeme zu entwickeln.</p><p>Wenn Sie mehr zum Thema NLP in unserem Blog lesen wollen, empfehlen wir den Artikel <a href="/wissens-hub/blog/natural-language-processing-basics" target="_blank"> NLP-Basics </a> . Allgemeine Informationen zum Thema KI haben wir auf <a href="https://www.codecentric.de/kuenstliche-intelligenz/" target="_blank"> codecentric.ai </a>  zusammengestellt.</p>

Kunden-E-Mails effizient verarbeiten – mit künstlicher Intelligenz

<p>Noch vor kurzer Zeit war E-Mail-Klassifikation mittels Deep Learning nur mit Spezialwissen und ausreichend Data Science Know-how möglich. Heute existieren sehr gute Open-Source-Bibliotheken mit fertigen Deep-Learning-Modellen, welche sehr weit optimiert sind und aktuellen Best Practices folgen. Mithilfe solcher Modelle kann man sofort und ohne viel Data Science Know-how Deep-Learning zur E-Mail-Klassifikation einsetzen.</p><p>Im Folgenden trainieren wir ein Deep-Learning-Modell der Open-Source-Bibliothek <a href="https://spacy.io/" target="_blank">spaCy</a> , um E-Mails zu klassifizieren. Das Beispiel ist so gedacht, dass es in einem Jupyter Notebook Schritt für Schritt nachvollzogen werden kann.</p><p>Warum es für Unternehmen hilfreich sein kann, sich mich E-Mail-Klassifikation zu beschäftigen, haben wir bereits im Blogpost <a href="/wissens-hub/blog/kunden-e-mails-effizient-verarbeiten-mit-kuenstlicher-intelligenz" target="_blank">Kunden-E-Mails effizient verarbeiten – mit Künstlicher Intelligenz </a>  beschrieben.</p><h2>Wie gehen wir vor?</h2><p>Unser Beispiel besteht aus vier Schritten:</p><h3>Daten beschaffen</h3><p>In der Praxis ist es häufig aufwändig einen Datensatz zu erstellen. Möchte man ein Modell zur Klassifikation von E-Mails aus einem spezifischen Unternehmenskontext trainieren, führt daran aber kein Weg vorbei. Neben dem Zusammenstellen der E-Mails gehört hierzu auch die Auswahl geeigneter Kategorien. Um brauchbare Ergebnisse erhalten zu können, benötigt man ausreichend (= mehrere Hundert) E-Mails pro Kategorie.</p><h3>Daten vorbereiten</h3><p>Typischerweise liegen die Daten nicht direkt in einem geeigneten Format vor. Daher ist es normalerweise notwendig, die Daten zunächst geeignet vorzubereiten. Ziel ist ein Datenformat, das einerseits statistische Analysen des Datensatzes erlaubt und mit dem andererseits leicht ein Modell trainiert werden kann.</p><h3>Modell trainieren</h3><p>Wir verwenden das in spaCy integrierte Deep-Learning Modell <code>textcat</code>. spaCy ist eine mächtige Open-Source Bibliothek für Python, die dafür gebaut wurde, Anwendungen im NLP-Bereich zu entwickeln. Um direkt loslegen zu können, beinhaltet spaCy bereits vortrainierte Sprachmodelle. Unsere Erfahrung zeigt, dass viele Use Cases bereits mit diesen Modellen umgesetzt werden können und es sich meistens nicht lohnt, ein eigenes Modell von Grund auf neu zu entwickeln.</p><h3>Modell evaluieren</h3><p>Abschließend überprüfen wir, wie gut das trainierte Modell neue E-Mails klassifiziert.</p><p>Fangen wir also an: Zunächst brauchen wir einen geeigneten Datensatz an E-Mails.</p><h2>Daten beschaffen</h2><p>Für unser Beispiel verwenden wir das öffentlich verfügbare <code>omqdata</code> Datenset, welches <a href="https://www.dfki.de/~neumann/resources/omqdata.html" target="_blank">hier</a>  heruntergeladen werden kann. Der Datensatz besteht aus E-Mails, die von Kunden an den Support eines Multimedia-Software-Unternehmens geschickt wurden.</p><p>Die E-Mails sind bereits anhand von Fehlermeldungen wie <code>"Fehlercode -9 beim Start des Programmes"</code> oder <code>"Altes Programm soll unter Windows 7 installiert werden"</code> in Kategorien eingeteilt. Verwandte Fehlermeldungen sind in Kategoriegruppen zusammengefasst. So sind zum Beispiel in Kategoriegruppe 6 die Fehlermeldungen <code>"Fehlercode -9 beim Start des Programmes"</code>, <code>"Fehler 9000 bei der Programminstallation"</code> und <code>"Programm startet nicht"</code> zu finden.</p><p>Im Folgenden gehe ich davon aus, dass der Datensatz heruntergeladen wurde und in einem Ordner entpackt vorliegt. Das XML-File <code>omq_public_interactions.xml</code> unseres Datensatzes enthält die E-Mails. Diese sind anhand von 41 unterschiedlichen Fehlermeldungen eingeordnet. Im XML-File <code>omq_public_categories.xml</code> befindet sich die Zuordnung der Fehlermeldungen in 20 Kategoriegruppen.</p><p><code>xml</code>-Files sind für die weiteren Schritte unhandlich. Wir bereiten daher im nächsten Schritt unsere Daten zur Weiterverarbeitung vor.</p><h2>Daten vorbereiten</h2><p>Um die Daten vorzubereiten, verwenden wir die Python-Module</p><div>

```python
import os
import pandas as pd
import numpy as np
import random
from bs4 import BeautifulSoup
```

</div><p>Außerdem bezeichnen wir mit <code>PATH_TO_DATA</code> den Pfad zum Ordner mit unserem Beispieldatensatz. Die E-Mails und Kategorien können dann mithilfe des Code-Blocks</p><div>

```python
def get_nice_text(item):
    text = item.getText()
    text = text.replace('\r',' ')
    text = text.replace('\n','')
    return text.strip()
 
with open(os.path.join(PATH_TO_DATA,'omq_public_interactions.xml'), 'rb') as f:
    soup = BeautifulSoup(f.read(), 'html.parser')
 
mails = soup.find_all("text")
categories = soup.find_all("category")
 
rows = []
for item, cat in zip(mails, categories):
    text = get_nice_text(item)
    categories = cat.getText().split(',')
    if len(categories) > 0:
        category = random.choice(categories)
    else:
        category = np.nan
    rows = rows + [[text, categories, category]]
 
df = pd.DataFrame(rows, columns=['text', 'categories', 'category'])
```

</div><p>geparsed und in einem Pandas Dataframe gespeichert werden.</p><p>Im Datensatz befinden sich neun E-Mails, die mehr als einer Kategorie zugeordnet sind. Für unser Beispiel haben wir für die neun E-Mails einfach zufällig eine der Kategorien ausgewählt. Dadurch liegt nun ein Datensatz vor, in dem jede E-Mail genau einer Kategorie zugeordnet ist.</p><p>In der Praxis kann Multiklassifikation sinnvoll sein. Kunden können beispielsweise mehr als ein Anliegen in einer E-Mail beschreiben. Grundsätzlich kann das spaCy Modell mit Multiklassifikation umgehen, dies würde aber den Rahmen unseres Beispiels sprengen.</p><p>Es ist oft hilfreich, die Daten als Pandas DataFrame vorliegen zu haben, um das Datenset zu explorieren und statistische Analysen zu machen. So kann man zum Beispiel über</p><div>

```python
df.categories.value_counts()
```

</div><p>sehen, wie viele E-Mails jeder Kategorie im Datensatz enthalten sind.</p><p>Grundsätzlich ist es sinnvoll, die Daten zu untersuchen, bevor man ein Modell damit trainiert. Die Analyse des Datensatzes ist aber nicht Gegenstand dieses Blogposts. Wer mehr über Datenanalyse und spaCy wissen möchte, kann zum Beispiel <a href="/wissens-hub/blog/natural-language-processing-basics" target="_blank">hier</a>  fündig werden.</p><p>Mit dem nachfolgenden Code-Block schreiben wir die Zuordnung der Kategorien in Kategoriegruppen in ein weiteres Pandas DataFrame.</p><div>

```python
with open(os.path.join(PATH_TO_DATA, 'omq_public_categories.xml'), 'rb') as f:
    soup = BeautifulSoup(f, 'html.parser')
 
categoryGroups = soup.find_all('categorygroup')
 
rows = []
for group in categoryGroups:
    group_id = group['id']    
    categories = group.find_all('category')
    cat_ids = []
    for cat in categories:
        cat_id = cat['id']
        cat_ids.append(cat_id)
        cat_text = get_nice_text(cat)
    rows = rows + [[group_id, cat_ids, cat_text]]
 
df_cat = pd.DataFrame(rows, columns=['group_id', 'cat_ids', 'cat_text'])
```

</div><p>Im letzten Schritt ordnen wir nun allen E-Mails die zugehörige Kategoriegruppe zu.</p><div>

```python
def get_encode_dic():
    encode_dic = {}
    for index, row in df_cat.iterrows():
        for cat_id in row.cat_ids:
            encode_dic[cat_id] = row.group_id
    return encode_dic
 
encode_dic = get_encode_dic()
df['category_group'] = df.category.map(encode_dic)
```

</div><p>Es ist eine gute Praxis, keine „Jupyter Notebook Monolithen“ zu bauen und das erste Notebook hier zu beenden. Die Ergebnisse der Datenvorbereitung können über den Befehl</p><div>

```python
df.to_pickle(os.path.join(PATH_TO_DATA, 'processed_mails.pkl'))
```

</div><p>gespeichert werden. Wir verwenden nun die <code>processed_mails</code> im nächsten Notebook, um ein Modell zu trainieren. Das Modell soll E-Mails der korrekten Kategoriegruppe zuordnen.</p><h2>Modell trainieren</h2><p>Wir verwenden die Python-Module</p><div>

```python
import os
import spacy
import pandas as pd
 
from spacy import displacy
from spacy.util import minibatch, compounding, decaying
 
from sklearn.model_selection import StratifiedShuffleSplit
```

</div><p>Zunächst laden wir mit dem Befehl</p><div>

```python
df = pd.read_pickle(os.path.join(PATH_TO_DATA, 'processed_mails.pkl'))
```

</div><p>die gerade vorbereiteten E-Mail-Daten in unser Notebook. Bevor wir mit dem Training des Modells beginnen, teilen wir den Datensatz via</p><div>

```python
X_all = df.text
y_all = df.category_group
 
seed = 42
sss = StratifiedShuffleSplit(n_splits=1, test_size=0.15, random_state=seed)
for train_index, test_index in sss.split(X_all, y_all):
    X_train, X_test = X_all.iloc[train_index], X_all.iloc[test_index]
    y_train, y_test = y_all.iloc[train_index], y_all.iloc[test_index]
```

</div><p>in Trainings- und Testdaten auf. So haben wir später noch E-Mails übrig, mit denen wir unser trainiertes Modell evaluieren können.</p><p>spaCy erwartet die eingehenden Trainingsdaten in einem bestimmten Format, in welches wir unsere Trainingsdaten via</p><div>

```python
cats = df.category_group.unique()
 
def get_catlist(y):
    cat_list = []
    for _, val in y.iteritems():
        categories = {'cats': {cat: 0.0 for cat in cats}}
        if val in cats:
            categories['cats'][val] = 1.0
        cat_list.append(categories)
    return cat_list
 
data_train = list(zip(X_train.values, get_catlist(y_train)))
```

</div><p>überführen können.</p><p>spaCy verfügt über ein vortrainiertes deutsches Sprachmodell, das wir im Folgenden verwenden möchten. Dieses muss zum Beispiel via</p><div>

```bash
python -m spacy download de_core_news_sm
```

</div><p>über die Konsole heruntergeladen werden. Danach laden wir mit</p><div>

```python
nlp = spacy.load('de_core_news_sm')
```

</div><p>das deutsche Sprachmodell in unser Notebook und aktivieren über</p><div>

```python
if 'textcat' not in nlp.pipe_names:
    textcat = nlp.create_pipe('textcat')
    nlp.add_pipe(textcat, last=True)
```

</div><p>das spaCy-Modell <code>textcat</code> zur Textklassifizierung. Als nächstes fügen wir über</p><div>

```python
for cat in cats:
    textcat.add_label(cat)
```

</div><p>die Kategorien unserer E-Mails hinzu. Jetzt können wir mit dem Training beginnen.</p><p>Da wir lediglich die Textklassifizierung trainieren und das Sprachmodell ansonsten nicht verändern möchten, schalten wir alles andere während des Trainings aus. Der folgende Code-Block trainiert <code>textcat</code> mit <code>n_iter</code> Durchläufen durch den Trainingsdatensatz.</p><div>

```python
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']
 
n_iter = 10
with nlp.disable_pipes(*other_pipes):  
    optimizer = nlp.begin_training()
    print("Training the model...")
    print('{:^5}'.format('LOSS'))
    for i in range(n_iter):
        losses = {}
        dropout = decaying(0.6, 0.2, 1e-4)
        batches = minibatch(data_train, size=compounding(1., 16., 1.001))
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(texts, annotations, sgd=optimizer, drop=next(dropout), losses=losses)
        print('{0:.3f}'.format(losses['textcat']))
```

</div><p>spaCy bietet verschiedene Möglichkeiten, Einfluss auf das Training zu nehmen und verschiedene fortgeschrittene Techniken zum Trainieren einzusetzen. Wer sich dafür interessiert, kann <a href="https://spacy.io/usage/training#tips" target="_blank">hier</a>  fündig werden. Je nach Projektkontext gilt es aber frühzeitige Optimierungen zu vermeiden. Ein solides Modell in Produktion ist oft besser als Ideen für ein optimiertes Modell im Jupyter Notebook.</p><h2>Evaluation</h2><p>Unser Modell können wir nun mithilfe der Testdaten evaluieren. Durch</p><div>

```python
def keywithmaxval(d):
    v=list(d.values())
    k=list(d.keys())
 
    return k[v.index(max(v))]
 
y_pred = []
for _, text in X_test.iteritems():
    doc = nlp(text)
    cat_pred = keywithmaxval(doc.cats)
    y_pred.append(cat_pred)
```

</div><p>erhalten wir die Vorhersagen unseres trainierten Modells für die E-Mails des Testdatensatzes. Nun können wir mittels</p><div>

```python
import matplotlib.pyplot as plt
import seaborn as sns;
from sklearn.metrics import classification_report, confusion_matrix
 
mat = confusion_matrix(y_test, y_pred, labels=cats)
plt.figure(figsize = (10,7))
sns.set(font_scale=1.2)
sns.heatmap(mat, square=True, annot=True, cbar=False, cmap="YlGnBu",annot_kws={"size": 13}, xticklabels=cats, yticklabels=cats)
plt.xlabel('Predicted label')
plt.ylabel('True label')
plt.show()
 
print(classification_report(y_test, y_pred, labels=cats))
```

</div><p>einerseits eine <a href="https://en.wikipedia.org/wiki/Confusion_matrix" target="_blank">Confusion Matrix</a>  und einen classification_report mit den Werten für Accuracy, Recall und F1-Score zu allen Kategorien erhalten. Anhand dieser Metriken kann man die Qualität des Modells in einer ersten Iteration ganz gut beurteilen.</p><p>Für unser vorliegendes Beispiel bekomme ich mit <code>n_iter = 25</code> die folgende Confusion Matrix.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/plud0f5qRMSQbigNN12r"><img width="250" height="248" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/plud0f5qRMSQbigNN12r" alt /></a> </p><p>Die Ergebnisse sind für den vorliegenden kleinen Datensatz schon ganz gut. So wird zum Beispiel die Kategoriegruppe 12 <code>"Problemen mit der Registrierung der Software"</code> recht zuverlässig erkannt.</p><p>Auffällig ist Kategoriegruppe 16 <code>"Kein Brenner gefunden"</code>. Hier wurden zwar alle vier im Testdatensatz enthaltenen E-Mails richtig erkannt, jedoch sind auch viele andere E-Mails fälschlicherweise dieser Kategoriegruppe zugeordnet. Um die Ergebnisse weiter zu verbessern gibt es verschiedene Möglichkeiten. Die wirksamste Möglichkeit ist bei Deep-Learning-Modellen in der Regel, dass man “einfach” mit einem größeren Datensatz trainiert. Insbesondere sind für einige Kategoriegruppen etwa von 1, 13, und 18 fast gar keine E-Mails im Datensatz vorhanden.</p><p>Für eine wirklich fundierte Beurteilung der Qualität des Modells reicht unser Beispieldatensatz leider nicht aus.</p><h2>Fazit</h2><p>In diesem Blogpost haben wir gesehen, dass es mithilfe von Open-Source-Bibliotheken wie spaCy heute möglich ist, ohne viel Data Science Know-how mit modernen Deep-Learning-Modellen E-Mails zu klassifizieren. Dabei wird sich der grundsätzliche Ablauf für einen anderen Datensatz nur minimal ändern.</p><p>Die Hürde für Softwareprojekte mit Deep-Learning-Anteil ist dadurch deutlich niedriger geworden. Es ist möglich, einen echten Mehrwert mit Deep-Learning zu erzeugen, ohne in einer Explorationsphase von mehreren Wochen mit viel Spezialwissen zunächst Modelle zu entwickeln und zu evaluieren.</p><p>Wer mehr über NLP erfahren möchte findet <a href="https://www.youtube.com/watch?v=GmLsb-o7hvM" target="_blank">hier</a>  ein passendes Video.</p><p>Mehr Informationen zum Thema Künstliche Intelligenz haben wir auf <a href="https://www.codecentric.de/kuenstliche-intelligenz/" target="_blank"> codecentric.ai </a>  zusammengestellt.</p>

E-Mail-Klassifizierung mit SpaCy

<p>Noch vor kurzer Zeit war der Einsatz von künstlicher Intelligenz (KI) nur mit großem Aufwand und ausreichend Spezialwissen möglich. Hauptsächlich große Internet-Konzerne wie Google, Apple und Facebook hatten das Geld, die Daten und die Expertise, um mittels KI innovative Produkte und Services zu entwickeln und damit Geld zu verdienen.</p><p>Inzwischen ist die Eintrittshürde für den Einsatz von KI erheblich gesunken. Aktuelle Entwicklungen, vor allem im Bereich von Cloud-Computing-Diensten, ermöglichen es nun auch kleinen und mittelständischen Unternehmen (KMU), kosteneffizient KI anzuwenden. Dabei heißt KI einzusetzen heutzutage nicht mehr zwingend in Forschung zu investieren, sondern sie in vielen Fällen schlicht anzuwenden.</p><p>Ein möglicher Anwendungsfall in diesem Kontext ist die KI-unterstützte (Teil-)Automatisierung der Qualitätskontrolle von Bauteilen basierend auf Bilderkennung, mit dem wir uns in diesem Artikel beschäftigen.</p><h2>Die Situation</h2><p>Der mittelständische ACME Zulieferungsbetrieb produziert täglich Bauteile in hoher Stückzahl und liefert diese in Großaufträgen an seine Kunden aus. Die Kunden von ACME legen großen Wert auf Qualität, und bereits wenige fehlerhafte Bauteile können zur Ablehnung eines gesamten Auftrags und damit zu großen Verlusten für ACME führen.</p><p>Aus diesem Grund hat ACME strenge Qualitätskontrollen etabliert. Hier werden unter hohem menschlichen Einsatz alle produzierten Bauteile manuell überprüft. Der Großteil der manuell überprüften Teile ist dank dieser Sorgfalt fehlerfrei. Die fehlerhaften Bauteile müssen dagegen häufig im Detail überprüft werden, da neben der Beurteilung auch der Grund für den Fehler gefunden werden muss. Für diese Begutachtung ist mehrjährige Erfahrung und dediziertes Domänen-Wissen notwendig.</p><p>Zur Kontrolle von Bauteilen und zur Qualitätssicherung beschäftigt ACME 53 Mitarbeiter*innen. Je weiter ACME es schafft, einfache Sachverhalte und Standardfälle automatisiert zu bearbeiten, desto mehr Zeit haben die Mitarbeiter*innen, um Spezialfälle zu beurteilen und Fehlerursachen zu verstehen.</p><p>ACME verfügt nicht über KI-Know-how. Die Firma hat auch kein Bestreben in ein ungewisses Feld wie KI-Forschung zu investieren und sieht auch keinen Bedarf eine Data-Science-Abteilung aufzubauen. Stattdessen möchte sich ACME nach wie vor darauf fokussieren, seinen Status als Hidden Champion auszubauen. Hierfür investiert ACME regelmäßig in aktuelle Zukunftstechnologie.</p><p>Durch Cloud-Computing-Dienste bietet sich für ACME eine gute Möglichkeit, moderne KI-Technologie einzusetzen und iterativ ein KI-System in der Qualitätskontrolle einzuführen – ganz ohne hohes Anfangsinvestment in KI-Forschung. Schritt für Schritt kann der Grad an Automatisierung weiter ausgebaut werden. So können perspektivisch nicht nur Fehler, sondern auch Fehlertypen automatisiert erkannt werden.</p><h2>Ein KI-System in der Qualitätskontrolle</h2><p>Das grundsätzliche Ziel für ein (teil-)automatisiertes System zur Qualitätskontrolle von Bauteilen basierend auf Bilderkennung ist es, den Automatisierungsgrad der Qualitätskontrolle schrittweise zu erhöhen. In der ersten Stufe sortiert das System neu produzierter Bauteile mittels einer KI-Komponente in die Klassen „okay“, „defect“ und „unclear“. Die Idee ist im folgenden Bild dargestellt.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GSzoG9Z9RJiEqDfFQb7O"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/GSzoG9Z9RJiEqDfFQb7O" alt="Teil-automatisiertes KI-System zur Qualitätskontrolle mit menschlicher Entscheidungshilfe bei unsicheren KI-Entscheidungen" width="346" height="223" /></a> <p>Teil-automatisiertes KI-System zur Qualitätskontrolle mit menschlicher Entscheidungshilfe bei unsicheren KI-Entscheidungen</p></div><p>In der Kategorie „unclear“ finden sich alle Bauteile, deren Qualität die KI nicht mit ausreichender Sicherheit bestimmen konnte. Das Maß an notwendiger Sicherheit lässt sich von außen vorgeben. So können z.B. nur solche Entscheidungen der KI akzeptiert werden, welche mit über 98 % Wahrscheinlichkeit getroffen wurden.</p><p>Nach dieser Beurteilung durch die KI sollten die mit „okay“ klassifizierten Bauteile perspektivisch ohne menschliche Eingriffe akzeptiert werden. Bauteile, welche durch die KI nur mit unzureichender Sicherheit klassifiziert und damit „unclear“ sind, müssen manuell nachbearbeitet werden. Die Bauteile in der Klasse „defect“ sind Gegenstand manueller Bearbeitung von Spezialist*innen. Hier liegt das Hauptaugenmerk auf Analyse des Fehlers und der Erforschung der Fehlerursachen.</p><p>Entscheidend für die Weiterentwicklung und Verbesserung des Systems ist dabei, dass alle Daten, die durch manuelle Bearbeitung entstehen, gesammelt werden. So eröffnet sich einerseits die Möglichkeit einen Feedback-Loop zu schaffen und die KI in regelmäßigen Abständen auf mehr Daten zu trainieren und so zu verbessern. Andererseits entsteht eine wertvolle Datenbasis, welche zukünftige weitere Ausbaustufen des Systems erlaubt.</p><h2>Vorgehensweise</h2><p>Es ist ratsam, zunächst mit einem abgegrenzten Use Case ohne zu viele Abhängigkeiten zu starten. Oft beobachten wir, dass in den ersten Feldversuchen die Ziele zu hoch gesteckt werden und man sich in der Komplexität der Aufgabe verliert. So ist es in der Praxis meistens unrealistisch, direkt in den ersten Schritten einen hohen Grad an Automatisierung anzustreben.</p><p>Anstatt sehr viel Zeit und Geld in die Entwicklung des „perfekten“ KI-Modells zu investieren, ist es häufig erfolgsversprechender, zunächst ein (teil-)automatisiertes System zu entwickeln. Dabei liegt der Fokus darauf das System tatsächlich in den bestehenden Prozess zu integrieren, um den Return on Investment schnellstmöglich zu erreichen. In dieser Phase treten typischerweise die eigentlichen Probleme auf, die gelöst werden müssen, bevor ein Mehrwert erzielt werden kann.</p><p>Grundlage für die Entwicklung ein erstes Basissystem sind die folgenden drei Kernpunkte:</p><ul><li>Daten beschaffen und labeln</li><li>Woher kommt die KI?</li><li>Integration in den Prozess</li></ul><p>Diese werden wir nun etwas genauer beleuchten.</p><h3>Daten beschaffen und labeln</h3><p>Ein entscheidender Schritt für KI-Systeme ist die Verfügbarkeit einer geeigneten Datenbasis. Um ein KI-System zur Qualitätskontrolle zu trainieren, werden gelabelte Bilder von Bauteilen benötigt. Für die erste Ausbaustufe sind das Bilder, die bereits in eine Kategorie „okay“ oder „defect“ eingeordnet sind. Für weitere Ausbaustufen etwa für die Erkennung von Fehlertypen wird eine ausreichende Anzahl von Bildern je Kategorie benötigt, um verlässliche Resultate erhalten zu können.</p><p>In vielen Fällen bildet die gelabelte Datenbasis den eigentlichen Wert des KI-Systems. So ist etwa ein gutes fachliches Verständnis der Domäne notwendig, um geeignete Kategorien und Gruppen von Fehlertypen festzulegen. Daher kann es für den ersten Schritt auch sinnvoll sein, das System ohne gut funktionierende KI zu integrieren und schlichtweg Daten zu sammeln.</p><h3>Woher kommt die KI?</h3><p>Die traditionelle Entwicklung eines KI-Modells ist häufig mit großen Schmerzen und Aufwand verbunden. Dies entspricht in vielen Fällen nach wie vor der Realität. Eigene KI-Modelle von Grund auf selbst zu entwickeln, ist in den meisten Fällen eine nicht zu unterschätzende Aufgabe. In der Praxis führt dieses Unterfangen viele Unternehmen in eine <a href="/wissens-hub/blog/wie-man-data-science-projekte-nicht-in-die-poc-sackgasse-manoevriert" target="_blank">Sackgasse</a>  und das damit verbundene Investment zahlt sich nicht aus.</p><p>Hier spielt der Teilbereich „Bilderkennung mit KI“ jedoch eine Sonderrolle, denn dieser gehört zu den am Besten gelösten Problemen im gesamten KI-Umfeld. So stellen heute viele Cloud-Anbieter bereits ausgereifte KI-Services wie zum Beispiel <a href="https://github.com/naxty/gcp-automated-quality-inspection" target="_blank">AutoML (Google Cloud)</a> , <a href="/wissens-hub/blog/bildklassifikation-mit-amazon-sagemaker" target="_blank">Sagemaker</a> <span><a href="/wissens-hub/blog/bildklassifikation-mit-amazon-sagemaker"> (Amazon Webservices)</a>  und <a href="https://azure.microsoft.com/de-de/services/cognitive-services/" target="_blank">Cognitive Services (Microsoft Azure)</a>  zur Verfügung. Dies eröffnet die Möglichkeit, aktuelle KI-Technologien zu verwenden, ohne in Forschung und Entwicklung investieren zu müssen und ohne tiefes Spezialwissen aufzubauen.</span></p><p>Auf diese Weise kann man mit fundiertem Basiswissen bereits in wenigen Schritten ein State-of-the-art-KI-Modell zur Bildklassifikation basierend auf einem eigenen Datensatz trainieren. Das resultierende KI-Modell ist für viele Anwendungsfälle „gut genug“, um einen Mehrwert zu erzielen und kann daher zur Integration in den Prozess verwendet werden.</p><p>Es ist grundsätzlich denkbar und oft auch möglich für konkrete Anwendungsfälle bessere Modelle zu entwickeln. Dies erfordert jedoch tiefgreifendes Spezialwissen und ist typischerweise mit erheblichem Aufwand und damit Kosten verbunden. Oft stehen hier Kosten und Nutzen in keinem sinnvollen Verhältnis.</p><p>Bei einem „eigenen“ Modell müssen zudem viele technische Themen wie z. B. das Deployment des Modells und Versionierung selbst gelöst werden, welche der Cloud-Anbieter in seinem Service out-of-the-box mitbringt.</p><h3>Integration in den Prozess</h3><p>Wenn mit einem KI-System die Qualitätskontrolle schrittweise automatisiert werden soll, besteht die eigentliche Herausforderung nicht darin ein KI-Modell zu entwickeln, sondern darin dieses System tatsächlich in den Gesamtprozess zu integrieren. Hierbei ist unbestritten, dass das System Fehler machen wird und typischerweise ist es auch mit einem KI-System nicht wirtschaftlich eine Fehlerrate von 0 % anzustreben. Ein KI-System bietet dagegen häufig die Möglichkeit, eine tolerierbare Fehlerrate effizienter zu erreichen.</p><p>Unserer Erfahrung nach ist dabei entscheidend, das System von Anfang an nicht mit dem Gedanken „Mensch kontrolliert Maschine“ oder „Mensch überprüft die Entscheidungen der Maschine“, sondern mit dem Grundsatz „Maschine entscheidet und der Mensch unterstützt den Prozess“ aufzusetzen, um die Effizienzgewinne mittelfristig auch tatsächlich realisieren zu können.</p><p>„Die Maschine entscheidet“ kann in diesem Fall auch heißen, Bauteile als „unclear“ zu klassifizieren und damit an einen Menschen zu übergeben. Entscheidend ist es, als „okay“ klassifizierten Bauteile so zu akzeptieren und eine etwaige Fehlerrate in Kauf zu nehmen.</p><p>Um die notwendige Akzeptanz der Entscheidungen KI-Systems zu gewährleisten und die Fehlerrate des Systems periodisch zu überprüfen, sind regelmäßige Benchmarks essenziell. Oft genügt es, den Benchmark nur mit der KI-Komponente digital durchzuführen. Dies kann unabhängig von der Produktion der Bauteile durchgeführt werden. Zudem liefern diese Tests Informationen darüber, wie der Schwellenwert für die Akzeptanz der KI-Entscheidungen im System justiert werden soll.</p><h2>Fazit</h2><p>In der Qualitätskontrolle bietet sich für viele mittelständische Unternehmen heute eine gute Möglichkeit, aktuelle KI-Technologie einzusetzen. KI ist nun in vielen Bereichen greifbar geworden, nicht zuletzt durch aktuelle Weiterentwicklungen von Public-Cloud-Diensten. Dadurch bedeutet die Verwendung von KI nicht mehr zwingend in Forschung zu investieren und tiefes Spezialwissen aufzubauen, sondern bewährte Technologie einzusetzen.</p><p>Die Nutzung von Cloud-Diensten senkt die Eintrittsbarriere erheblich und ermöglicht es insbesondere nicht Software-affinen Unternehmen, sich auf ihre Wertschöpfungskette zu konzentrieren. Auf diese Weise kann KI in vielen Fällen zu einem sinnvollen Investment in aktuelle Zukunftstechnologien werden.</p><p>Unserer Herangehensweise an KI-Projekte haben wir in unserem <a href="https://www.codecentric.de/wissens-hub/webinare/webinar-data-science-projekte" target="_blank">On-Demand-Webinar (Deutsch)</a>  vorgestellt.</p><p>Jetzt einen unverbindlichen Austausch anfordern über <a href="mailto:ki@codecentic.de">ki@codecentic.de</a> .</p>

KI für KMU: (Teil-)Automatisierung der Qualitätskontrolle von Bauteilen

<p>Warum gelingt es Data-Science-Initiativen häufig nicht, einen echten Mehrwert zu schaffen? Wir haben einige Ursachen dafür ausgemacht. In diesem Blogpost stellen wir vier typische Fallen für Data-Science-Projekte vor und geben Tipps, wie Du sie umschiffen kannst.</p><p><strong>Hinweis: Dieser Artikel erschien ursprünglich im Softwerker-Magazin Vol. 14. <a href="https://www.codecentric.de/wissens-hub/softwerker/softwerker-14" target="_blank">Das gesamte Heft kann hier als PDF heruntergeladen werden</a> .</strong></p><p>Seien wir ehrlich: Obwohl der <a href="https://blog.codecentric.de/tag/ki/">Künstliche-Intelligenz</a> -Hype aktuell überall zu spüren ist, gelingt es den meisten Unternehmen derzeit noch nicht, mit ihren Data-Science-Initiativen einen Mehrwert für sich oder ihre Kunden zu schaffen. Im Gegenteil: Wir erleben oft, dass sehr viel Budget in Data-Science-Projekte investiert wird, aber nach dem x-ten Proof of Concept (PoC) noch kein Geld verdient wurde.</p><p>In vielen Fällen beobachten wir sogar, dass Data-Science-Projekte nicht fortgeführt werden und stattdessen abrupt enden, obwohl es eigentlich erfolgreiche Proofs of Concept gab. Bevor der erste Euro verdient wurde, wird stattdessen häufig schon ein weiterer Proof of Concept gestartet. So hangeln sich die Data Scientists von PoC zu PoC und werden irgendwann mit der unangenehmen Wahrheit konfrontiert, dass trotz aller Bemühungen kein Business Value geschaffen wurde. Da auch wir und unsere Data Scientists regelmäßig an solchen Projekten beteiligt sind, sind wir diesem Problem auf den Grund gegangen und möchten im Folgenden einige Erkenntnisse teilen.</p><h2>Abgrenzung</h2><p>Der Artikel fokussiert sich auf die Verprobung einer Idee im Data-Science-Kontext, mit dem Ziel, daraus ein Produkt zu entwickeln und damit Geld zu verdienen. Neben diesem Fokus kann es für Data-Science-Projekte auch das Ziel geben, neue Technologien kennenzulernen und deren Potenzial abzuschätzen. Dabei steht eventuell zunächst nicht die kommerzielle Nutzung dieser Technologie im Vordergrund. Solche Projekte benötigen in der Regel eine andere Steuerung, etwa über die Begrenzung des Budgets oder der Zeit für das Projekt.</p><h3>Quellen</h3><p>Während wir die nachfolgend beschriebenen Situationen in unserem Projektalltag regelmäßig antreffen, basieren die empfohlenen Lösungsansätze auf Ideen und Konzepten verschiedenster Quellen im Kontext von „Lean Product Development“. Maßgeblich inspiriert wurde der Artikel insbesondere durch die Arbeiten, die in den Referenzen <a href="https://developers.google.com/machine-learning/guides/rules-of-ml/" target="_blank">[1]</a>  <a href="https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf" target="_blank">[2]</a>  <a href="http://theleanstartup.com/book" target="_blank">[3]</a>  <a href="https://multithreaded.stitchfix.com/blog/2019/03/11/FullStackDS-Generalists/" target="_blank">[4]</a>  <a href="http://lpd2.com/" target="_blank">[5]</a>  aufgeführt sind (s. Artikelende).</p><h2>Fokus auf den Business Case</h2><p>Viele Data-Science-Projekte starten ohne ausreichend geschärften Business Case. Das ist zunächst einmal nicht unbedingt schlimm. Problematisch kann es jedoch werden, wenn der Business Case in der Anfangsphase des Projekts nicht weiter verfeinert und geschärft wird. Idealerweise unterstützt bereits der PoC dabei, den eigentlichen Business Case zu validieren.</p><p>Ein Beispiel: Die Churn Rate beschreibt den Anteil der Bestandskunden eines Unternehmens, die innerhalb eines Zeitfensters das Unternehmen verlassen. Eine Churn Rate von 2 % pro Monat bedeutet zum Beispiel, dass das Unternehmen jeden Monat 2 % seiner Bestandskunden verliert. Um gesund zu bleiben, ist es für das Unternehmen sehr wichtig, diese Churn Rate zu kompensieren – etwa durch Neukundengewinnung. Typischerweise ist das Gewinnen neuer Kunden jedoch deutlich teurer, als Bestandskunden zu halten. Um gezielt diejenigen Kunden anzusprechen, die für das Unternehmen interessant sind, aber potentiell abwandern, ist daher Churn Prediction ein weit verbreiteter Anwendungsfall.</p><p>Nehmen wir an, wir möchten validieren, ob es sinnvoll ist, Churn Prediction zu betreiben. Einem ersten Impuls folgend, wird ein PoC mit der Frage  „Können wir Churn vorhersagen?” oder „Wie gut können wir Churn vorhersagen?“ gestartet. Zur Beantwortung der ersten Frage benötigt man streng genommen keinen PoC, denn die Antwort ist schlicht: „Ja!“. Die zweite Frage ist schon besser, führt aber immer noch etwas an der eigentlichen Kernfrage „Kann ich mit Churn-Prediction einen Mehrwert für das Unternehmen schaffen?“ vorbei.</p><p>In einem PoC herauszufinden, dass man für 40 % aller Kunden Churn korrekt vorhersagen kann, ist nur ein Teilaspekt, denn eine Genauigkeit von 40 % kann je nach Kontext gut oder schlecht sein. Um wirklich eine Vorstellung davon zu entwickeln, wie die 40 % einzuschätzen sind, muss man den zugrundeliegenden Business Case genauer untersuchen. Dazu gehört es einerseits, konkrete Maßnahmen zu finden, mit denen man potentiellen Churn möglicherweise verhindern kann. Andererseits müssen konkrete Annahmen zu Kosten und Nutzen dieser Maßnahmen getroffen und validiert werden.</p><p>Ein Ziel des PoCs wäre es dann, konkrete Annahmen darüber zu treffen, was eigentlich passieren muss, dass mit der zugrundeliegenden Idee – z. B. Churn Prediction  –  ein Mehrwert geschaffen werden kann. Ein Schritt, der in diese Richtung führt, ist es, statt lediglich die Genauigkeit des Modells zu betrachten, eine konkrete Business-Metrik zu entwickeln. Die Genauigkeit des Modells oder andere Modelleigenschaften spielen hierbei natürlich eine Rolle, sind aber nur Faktoren neben Größen wie „Kosten für Neukundengewinnung“ oder „Kosten für Gutscheine, um die Kunden zu behalten“, um den Business Case zu beurteilen.</p><p>In der Anfangsphase ist es dazu häufig notwendig, pragmatische Annahmen für verschiedene Faktoren zu treffen. Im Beispiel Churn Prediction ist es etwa sehr schwer vorherzusagen, wie viele Kunden aufgrund eines Gutscheins letztlich nicht kündigen werden. Dennoch spielt dieser Anteil eine große Rolle für den Erfolg der Idee und es ist wichtig, diese Größe in einer Business-Metrik transparent zu machen. Das Ziel ist es, eine initiale Metrik zu entwickeln, die auf konkreten Annahmen beruht. Basierend auf neuen Erkenntnissen und im Projektverlauf kann die Metrik dann stets iterativ verfeinert und die Annahmen können validiert werden.</p><p>Ohne Fokussierung auf den Business Case wird häufig zu spät oder gar nicht in Betracht gezogen, was überhaupt alles passieren muss, um tatsächlich Geld mit der Ausgangs-Idee zu verdienen. Oft beobachten wir, dass diese Fragen bewusst zum Projektstart ausgeklammert werden. In vielen Fällen führt diese Unklarheit aber letztlich dazu, dass die Data-Science-Initiative nach dem PoC nicht fortgeführt wird. Es fehlt eine Vision für das Produkt, um der Data-Science-Initiative den nötigen Vorschub zu leisten.</p><h2>Mindset zum Projektbeginn</h2><p>Es macht sehr viel Spaß, sich mit KI und neuronalen Netzen zu beschäftigen. Fast täglich werden wir mit neuen Möglichkeiten konfrontiert, die uns KI eröffnet. Darum ist es sehr wichtig, sich ständig daran zu erinnern, dass KI und andere Technologien im Kontext von Data-Science-Projekten lediglich ein Hilfsmittel dafür sind, einen Mehrwert für Kunden zu schaffen.</p><p>Startet ein Data-Science-Projekt mit einem Proof of Concept und der oben beschriebenen Fragestellung:  „Wie gut können wir Kündigungen vorhersagen?“ – oder allgemeiner: „Wie gut ist es möglich, mit KI XYZ zu erreichen?“ – ist diese sehr interessant und es ist äußerst spannend, sich mit der Frage zu beschäftigen. Ein solcher Projektstart führt jedoch regelmäßig dazu, dass im PoC exzessive Modelloptimierung betrieben wird.</p><p>In der Anfangsphase des Projekts ist es meistens nicht nötig, das vermeintlich beste Modell zu entwickeln. Dennoch werden häufig direkt am Anfang sehr viele Ressourcen in die Modelloptimierung investiert. Die folgenden Gründe sprechen gegen dieses Vorgehen:</p><ul><li>Ein Modell kann beliebig lange weiteroptimiert werden und es gibt oft einen unerschöpflichen Ideenpool, wie das Modell weiter verbessert werden könnte.</li><li>Der Aufwand zur Verbesserung des Modells wächst typischerweise exponentiell und es ist oft noch nicht klar, ob eine Verbesserung der Genauigkeit von 84 auf 87 % wirtschaftlich und überhaupt notwendig ist.</li><li>Neben einem Modell benötigt man noch eine Vielzahl weiterer Software-Komponenten, etwa zum Deployen des Modells und zur Auslieferung der Vorhersageergebnisse, um tatsächlich einen Mehrwert zu generieren.</li></ul><p>Pragmatisch gesehen benötigt man, um weiter zu kommen, im ersten Schritt lediglich ein Modell, das schlicht gut genug ist. Die Beurteilung von „gut genug“ kann anhand der oben beschriebenen Business-Metrik erfolgen. Modell-Metriken sind dabei nur ein Faktor in der eigentlichen Business-Metrik und oft entscheiden letztlich ganz andere Faktoren darüber, ob sich mit der Idee überhaupt Geld verdienen lässt.</p><p>Eine Empfehlung ist es daher, bereits vom ersten Projekttag an das Gesamtbild nicht aus dem Blick zu verlieren und sich regelmäßig die Frage zu stellen:  „Ist das, was ich gerade vorhabe, notwendig, um herauszufinden, ob ich mit der Idee einen Mehrwert schaffen kann?”</p><h2>Schnell Erfahrung sammeln</h2><p>Zu einem gewissen Zeitpunkt befindet sich das Projekt hoffentlich in der Situation, dass ein erstes Modell vorliegt und die Business-Metrik einen relevanten Mehrwert erwarten lässt. Hier basiert die Business-Metrik häufig zwar nicht auf validierten Annahmen, dennoch ist an dieser Stelle ein entscheidender Punkt im Projekt erreicht: Spätestens jetzt geht es darum, damit zu beginnen, die Annahmen hinter dem Business Case zu validieren.</p><p>Eine mögliche Falle, in die das Projekt in dieser Situation geraten kann, sind offene Diskussionen über Vermutungen, wie das Modell noch weiter verbessert werden kann oder Hypothesen über den zugrundeliegenden Business Case. Der Grund dafür ist, dass die Diskussionen in vielen Fällen schlicht zu keinem Ziel führen können. Das liegt daran, dass die relevantesten Erkenntnisse nur noch mit Feedback von echten Nutzern und Daten erlangt werden können.</p><p>Um möglichst schnell von hypothetischen „Was wäre, wenn“-Diskussionen zu „Was ist, wenn“-Erkenntnissen zu gelangen, gilt es daher, die gesamte Energie darauf zu konzentrieren, einen ersten Prototyp zu bauen und diesen unter realen Bedingungen zu testen. Hierbei geht es darum herauszufinden, ob sich der erwartete Mehrwert tatsächlich realisieren lässt. Also unter anderem die Annahmen der Business-Metrik zu validieren und zu testen, ob das Modell auf echten Daten ebenso zuverlässig und genau funktioniert wie auf den Beispieldaten.</p><p>Ein entscheidender Aspekt hierbei ist, so schnell wie möglich ein tragfähiges Fundament für den Business Case zu schaffen. Je länger die Validierung der Annahmen verzögert wird, desto länger basieren Entscheidungen und Weiterentwicklungen auf nicht validierten Annahmen. Dies führt in der Regel dazu, dass die Geschwindigkeit, mit der Anpassungen vorgenommen werden können, immer weiter sinkt und Letztere zusätzlich immer teurer und damit schmerzhafter werden.</p><p>Außerdem treten – sobald es darum geht, einen Prototyp zu bauen, diesen in eine IT-Infrastruktur zu integrieren und ihn mit Kunden zu testen – regelmäßig technische Herausforderungen auf, die bis dahin oft nicht absehbar waren. Manchmal verändern diese Herausforderungen den gesamten Business Case und in seltenen Fällen scheitert die gesamte Idee etwa daran, dass etwa Aspekte des Datenschutzes nicht erfüllt werden können.</p><p>Gelingt die Integration des Prototyps, stellt sich bei der Validierung typischerweise schnell heraus, dass der Business Case nicht von ein paar Prozent Modellgenauigkeit abhängt, sondern andere Faktoren einen viel größeren Einfluss haben. Umso schmerzlicher ist es, wenn diese Lernerfahrungen erst viel später gemacht werden und stattdessen zunächst viele Ressourcen in die Modelloptimierung investiert wurden.</p><p>Unsere Empfehlung ist es daher, möglichst schnell nicht mehr „discussion-driven“, sondern „experiment-driven“ zu arbeiten und validierte Erkenntnisse über den Business Case zu sammeln.</p><h2>Zusammenstellung des Projektteams</h2><p>Wenn man die bisher genannten Punkte im Projekt berücksichtigen möchte, zeigt sich schnell, dass hierfür eine Vielzahl unterschiedlicher Fähigkeiten, Kenntnisse und Erfahrungen nötig ist.</p><p>Data-Science-Projekte werden allerdings häufig mit einem Team gestartet, in dem ein stark akademisch geprägtes Mindset überwiegt. Dies führt regelmäßig dazu, dass nicht der Kundennutzen oder der Business Case im Vordergrund steht, sondern die Begeisterung für KI und neuronale Netze und damit die Technologie. Das wiederum fördert einen Fokus auf Modelloptimierung und eine große Bereitschaft im Team, den zugrundeliegenden Business Case nicht zu hinterfragen.</p><p>Oft haben Data Scientists keinen Hintergrund in Softwareentwicklung. Das allein ist zunächst kein Problem, da Data-Science-Expertise als solche wichtig für das Projekt ist. Besteht das Team jedoch überwiegend aus Data Scientists ohne Software-Engineering-Hintergrund, stellt die Entwicklung eines Prototyps das Team in vielen Fällen vor eine schwer lösbare Aufgabe und es ist schlicht nicht zu erwarten, dass ein solches Team einen schlanken Prototyp entwickelt, der schnell iterativ angepasst werden kann.</p><p>Fehlt die Expertise für Produktentwicklung im Team, gelingt es in vielen Fällen nicht, eine belastbare Produktvision zu erarbeiten. Dadurch verschiebt sich häufig der Fokus im Laufe des Projekts und verlagert sich vom Ziel, Kundennutzen schaffen, hin zu anderen technischen Zielen wie Modelloptimierung. Dies führt regelmäßig dazu, dass die Idee nach dem Nachweis der technischen Machbarkeit nicht weiterverfolgt wird und nach einem Abschlussvortrag in der Schublade verschwindet.</p><p>Nach unserer Erfahrung benötigt das Team bereits von Anfang an Expertise im Bereich Data/Software Engineering und Produktentwicklung. So kann es einerseits gelingen, den Übergang von der Validierung des Business Case zur Entwicklung eines Prototyps ohne unüberwindbare Reibungseffekte zu gestalten, da Themen wie Data Pipelines, Deployment des Modells und Auslieferung der Ergebnisse an die Nutzer direkt mitgedacht werden können. Andererseits kann aktiv an der Produktvision gearbeitet werden und der fachliche Input zur Erstellung einer Business-Metrik ist im Team vorhanden. Letztlich geht es darum, das Team so aufzustellen, dass der Fokus auf dem eigentlichen Ziel liegt: mit einem neuen Produkt Geld zu verdienen.</p><p>Wir empfehlen daher, das Team so zu gestalten, dass es alle notwendigen Fähigkeiten besitzt, um dieses primäre Ziel zu erreichen. Als zugrundeliegendes Paradigma gilt hierbei, alle Rahmenbedingungen so zu setzen, dass möglichst schnelle Feedback-Zyklen möglich sind. Dafür ist es insbesondere notwendig, dass das Team alle Entscheidungen im Projektkontext selbst treffen kann. Dazu gehören einerseits technische Entscheidungen, aber auch Entscheidungen zur Produktvision und die Möglichkeit, direkten Kontakt zu Kunden herzustellen, um Feedback einholen zu können.</p><h2>Fazit</h2><p>Typischerweise entsteht der größte Mehrwert für Kunden beim ersten Schritt von keiner Lösung zu einer Lösung. Um den Schritt mit dem Kunden zu gehen, muss die Lösung nicht optimal, sondern lediglich gut genug sein. In diesem Artikel haben wir aufgezeigt, welche Rahmenbedingungen in Data-Science-Projekten regelmäßig dazu führen, dass der Schritt zum Kunden sehr spät oder gar nicht gegangen wird. Da fehlendes Kundenfeedback häufig zu vorschnellen und kostenintensiven Optimierungen führt und nicht validierte Annahmen schlicht bedeuten, zu lange und aufwändig am Kunden vorbei zu entwickeln, ist es hilfreich, sich regelmäßig auf das primäre Ziel des Projekts zu fokussieren.</p><p>Der Artikel liefert keine Blaupause für Data-Science-Projekte; er zeigt lediglich auf, welche Bedingungen problematisch sein können, wenn es darum geht, das Ziel – ein erfolgreiches Data-Science-Produkt – zu erreichen. Bei der Umsetzung konkreter Projekte sind unterschiedlichste Rahmenbedingungen vorhanden, die verschiedenste Anpassungen der Vorgehensweise verlangen. Oft ist es nur sehr schwer oder gar nicht möglich, diese Rahmenbedingungen zu verändern. Das Ziel dieses Artikels war es daher, eine generelle Sensibilität zu schaffen, um im nächsten Projekt ein paar hinderliche Rahmenbedingungen gezielt hinterfragen zu können.</p><p>Unsere Herangehensweise an Data-Science-Projekte präsentieren wir in unserem kostenlosen <a href="https://www.codecentric.de/wissens-hub/webinare/webinar-data-science-projekte" target="_blank">On-Demand-Webinar (Deutsch)</a> . Außerdem zeigen wir in unserem <a href="/wissens-hub/blog/dish-o-tron-no-more-dirty-dishes-ai" target="_blank">Deep-Learning Tutorial: DISH-O-TRON (Englisch)</a>  (mit einem Augenzwinkern) wie man von einer Data-Science-Idee bis zum Ende-zu-Ende Prototypen kommt.</p><p>Jetzt einen unverbindlichen Austausch anfordern über <a href="mailto:ki@codecentic.de">ki@codecentic.de</a> .</p><h3>Referenzen & Link-Tipps</h3><p>[1] <a href="https://developers.google.com/machine-learning/guides/rules-of-ml/" target="_blank">https://developers.google.com/machine-learning/guides/rules-of-ml/</a> <br /> [2] <a href="https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf" target="_blank">https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf</a> <br /> [3] <a href="http://theleanstartup.com/book" target="_blank">http://theleanstartup.com/book</a> <br /> [4] <a href="https://multithreaded.stitchfix.com/blog/2019/03/11/FullStackDS-Generalists/" target="_blank">https://multithreaded.stitchfix.com/blog/2019/03/11/FullStackDS-Generalists/</a> <br /> [5] <a href="http://lpd2.com/" target="_blank">http://lpd2.com/</a> <br /> [6] <a href="/wissens-hub/blog/codecentric-ai-bootcamp">codecentric.AI Bootcamp</a> <br /> [7] <a href="https://info.codecentric.de/data-lunch" target="_blank">Data Science zum Mittagessen</a> </p>

Wie man Data-Science-Projekte nicht in die PoC-Sackgasse manövriert

KI in der Praxis: Fehlerhafte Bauteile mit AutoML in der Google Cloud identifizieren

<p>Heutzutage steht fast alles, was mit den Labels „künstliche Intelligenz (KI)“ oder „Machine Learning (ML)“ versehen ist, für Fortschritt. Seltsamerweise schließt diese Assoziation jedoch häufig die Themen Daten und Dateninfrastruktur nicht ausreichend mit ein, und das, obwohl ML-Systeme typischerweise in hohem Maße von einer geeigneten Dateninfrastruktur abhängig sind. Um tatsächlich Mehrwert mit einem ML-System zu erzeugen, sind insbesondere (automatisierte) Daten-Pipelines erforderlich, die das System solide in die IT-Landschaft integrieren.</p><p><strong>Hinweis: Dieser Artikel erschien ursprünglich im Softwerker-Magazin Vol. 16. <a href="https://www.codecentric.de/wissens-hub/softwerker/softwerker-16" target="_blank">Das gesamte Heft kann hier als PDF heruntergeladen werden</a> .</strong></p><h2>Eine Wahrheit über Data-Science-Projekte</h2><p>Betrachten wir die folgende Situation: Ein Data-Science-Projekt startet mit einem Proof of Concept (PoC) und einem versierten Projektteam, das direkt anfängt, in schnellen Iterationen Erkenntnisse zu liefern. Nach Ablauf der Timebox des PoC ist es tatsächlich gelungen, vielversprechende Ergebnisse zu erzielen. So gibt es die Gelegenheit, diese Ergebnisse vor „wichtigen Entscheidungsträger*innen“ zu präsentieren. Nachdem das Projektteam die Präsentation mit Bravour gemeistert hat, wird immer deutlicher: Die Idee ist ein Volltreffer! Alle fühlen sich großartig.</p><p>Unter der Haube wurden die vielversprechenden Data-Science-Ergebnisse jedoch auf einem lokalen Rechner mit Daten-Dumps (CSV-Dateien oder Excel-Tabellen) aus mehreren Datenquellen erzielt. Die Daten-Dumps wurden vorab von einer SQL-Expertin mit einer speziellen Abfolge magischer SQL Queries sorgfältig von Hand erzeugt. Dieses Vorgehen ist oft richtig, und es ist sehr sinnvoll, zunächst mit Daten-Dumps zu arbeiten, um schnell Erkenntnisse zu erlangen. Das beschriebene Vorgehen ist aber auch zweifellos eine Abkürzung.</p><p>Die Abkürzung wird dann zum Problem, wenn sie nicht transparent ist. Noch schlimmer, wenn sie nicht so kommuniziert ist, dass der zugrunde liegende Trade-off von „den Entscheidungsträger*innen“ verstanden wurde. Denn der Trade-off für die schnellen Einsichten und Ergebnisse während des PoC besteht in der Regel darin, weder die zugrunde liegende Dateninfrastruktur noch den Aufbau geeigneter Daten-Pipelines während des PoC anzugehen. Spätestens nachdem nun alle die erfolgreiche Präsentation gesehen haben und jetzt sofort einen Mehrwert innerhalb weniger Tage erwarten, müssen wir der Wahrheit ins Auge sehen: Wir haben eine Abkürzung genommen. Bevor wir tatsächlich Mehrwert generieren können, müssen wir noch einiges nachlegen.</p><h2>Eine schwere Bürde für Data-Science-Projekte</h2><p>Da es, wie in unserer fiktiven Geschichte skizziert, möglich ist, kurzfristige, aber sichtbare Erfolge zu erzielen, treibt der KI-Hype Unternehmen oft dazu, eine Reihe von Data-Science-Projekte zu starten. Jedoch wird häufig nicht zeitgleich ein Bewusstsein dafür geschaffen, welche Implikationen der (gewünschte) Produktiveinsatz eines ML-Systems für die Dateninfrastruktur hat. Klar ist: Um Ideen zu validieren, müssen die potentiell bestehenden Unzulänglichkeiten der Infrastruktur nicht alle zuerst gelöst werden. Ein Problem entsteht aber regelmäßig dann, wenn sich nicht alle Beteiligten darüber im Klaren sind, dass die Entwicklung eines Ende-zu-Ende-Prototyps oder gar eines produktionstauglichen ML-Systems – je nach vorhandener Dateninfrastruktur – ein sehr anspruchsvoller nächster Schritt nach einem PoC sein kann.</p><p>Im schlimmsten Fall führt dies zu der bizarren Situation, dass ein Data-Science-Projektteam nach dem PoC vor der Aufgabe steht, die unternehmensweiten Herausforderungen in Sachen Daten und Dateninfrastruktur ganz nebenbei mit zu lösen. Bei dem (verzweifelten) Versuch, Mehrwert zu schaffen, sieht sich das Projektteam dabei mit unlösbaren Aufgaben konfrontiert und es gibt wenig oder gar kein Verständnis für die Schwierigkeiten des Teams bei den Stakeholdern und damit häufig auch keine Unterstützung. Dies führt regelmäßig zu Frustration beim Projektteam und letztlich zum Scheitern der einst so vielversprechenden Idee, da kein Mehrwert in der erwarteten Zeit erzeugt werden konnte.</p><p>Im übertragenen Sinne versucht das Team, einen Sportwagen (das State-of-the-Art-KI-Modell) auf einem Waldweg (der Dateninfrastruktur) zu fahren und alle Außenstehenden fragen sich, warum das Team nicht einfach in voller Geschwindigkeit fährt. So entsteht häufig der Eindruck, dass es einfach nicht möglich ist, mit Data-Science-Projekten tatsächlich einen Mehrwert zu generieren.</p><h2>Daten – eine Frage der Einstellung</h2><p>Historisch gesehen ist der Aufbau einer neuen Dateninfrastruktur mit großen Schmerzen für Unternehmen verbunden. Dies liegt daran, dass sie typischerweise mit enormen Investitionen von Zeit und Geld (und Nerven) verbunden ist. Darüber hinaus gibt es verschiedene Architekturen, z. B. Data Lakes (siehe <a href="https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/" target="_blank">hier</a>  und <a href="https://jamesdixon.wordpress.com/2014/09/25/data-lakes-revisited/" target="_blank">hier</a> ), <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html" target="_blank">Data Lakehouses</a> , <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html" target="_blank">Data Meshes</a>  und natürlich das <a href="https://www.oracle.com/de/database/what-is-a-data-warehouse/" target="_blank">klassische Data Warehouse</a>  mit jeweils unterschiedlichen Vorteilen und Kompromissen, die es zu berücksichtigen gilt.</p><table><tbody><tr><th><strong>Ein kleiner Exkurs: Data Warehouse, Data Lake, Data Mesh, Data Lakehouse</strong></th></tr><tr><td> <p>Das zentrale <strong>Data Warehouse</strong> wird klassisch zur Konsolidierung großer, strukturierter Datenmengen aus verschiedenen Quellen verwendet und deckt einen Großteil der Einsatzzwecke etwa aus dem Reporting und Controlling sehr gut ab. Die typischerweise darin enthaltenen strukturierten Daten sind jedoch einerseits oft ohne nennenswerte Historie und andererseits stark aggregiert und daher regelmäßig nicht ausreichend für Data-Science-Anwendungsfälle.</p><p>Das dauerhafte Speichern großer Datenmengen in einer Data-Warehouse-Lösung ist meistens sehr teuer und damit unwirtschaftlich; hier liefert ein <strong>Data Lake</strong> einen möglichen Ausweg. Die Idee ist es, die großen (unstrukturierten) Datenmengen – wie z. B. Sensordaten – für potentielle Data-Science-Anwendungsfälle der Zukunft in günstigen Speicherformaten (oft in der Cloud) vorzuhalten. In Kombination mit passenden Data-Processing-Tools kann auf diese Weise ein Self- Service geschaffen werden, der es unterschiedlichen Nutzergruppen erlaubt, eigenständig und flexibel benötigte Daten nach ihren jeweiligen Wünschen zu untersuchen und passend weiterzuverarbeiten.</p><p>In der Praxis kann der Data-Lake-Ansatz jedoch zu einem „Dump and forget“ aller verfügbaren Daten in den Data Lake führen. Damit ist es für potentielle Konsumenten der Daten – welche die Daten ggf. erst mit einer Verzögerung von Monaten oder Jahren verwenden wollen – häufig schwer bis unmöglich, die Daten ohne substantielle Hilfe selbständig zu verwenden. Unter anderem an diese Herausforderung knüpft die Idee des <strong>Data Meshes</strong> an. Hier wird u. a. die Verantwortung für die Qualität und Nutzbarkeit der Daten an die Produzenten der Daten übertragen. Eine Aufgabe dabei ist es, insbesondere den Wahrheitsgehalt und die Vertrauenswürdigkeit der Daten zu garantieren. So erhalten die Daten den Stellenwert eines (internen) Produkts, das den Usern (Konsumenten) möglichst anwenderfreundlich zur Verfügung gestellt wird.</p><p>Orthogonal dazu steht die Idee des <strong>Lakehouse</strong> mit dem Ziel, die Konzepte von Data Lake und Data Warehouse zu einem gemeinsamen System zu verschmelzen. Das Zielsystem soll einerseits schnelle Exploration und Validierung potentieller Data-Science-Ideen unterstützen, aber andererseits auch ETL-Strecken von den Rohdaten bis zur aggregierten und strukturierten Sicht für Data Warehouses ermöglichen. Dabei setzen alle Anwendungen auf derselben Infrastruktur und insbesondere einer gemeinsamen Datenbasis auf.</p></td></tr></tbody></table><p>Den eigentlichen Kern und damit das Fundament zur Realisierung der durch eine der Architekturen in Aussicht gestellten Verbesserung sind jedoch ausdrücklich nicht die technischen Feinheiten. Denn im Kern geht es um die Frage, welchen Stellenwert Daten im Unternehmen haben und insbesondere darum, den Umgang und die Bedeutung mit und von Daten im Unternehmen zu verändern. Dieser Wandel ist wahrscheinlich der wichtigste Schritt bei der Umsetzung der Architekturen und damit entscheidend dabei, mit Data-Science-Projekten dauerhaft Mehrwert zu generieren.<br /> Die Erkenntnis, dass die Etablierung einer geeigneten „Data Culture“ im Unternehmen den zentralen Schritt in Richtung Mehrwert mit ML-Systemen darstellt, ist zwar nicht neu, aber nach unserer Beobachtung noch nicht ausreichend in der Breite adaptiert.<br /> Dabei gibt es jedoch (mindestens) zwei Gründe, die für ein Umdenken beim Thema Daten sprechen.</p><h4>1. Schnellere Zykluszeit für die Validierung von Ideen</h4><p>Es liegt auf der Hand, dass ein PoC (wie oben beschrieben) schnell erste Erkenntnisse darüber liefert, ob eine Idee vielversprechend ist oder nicht. Ohne eine geeignete Dateninfrastruktur ist es jedoch in der Regel schwierig und kostspielig, eine Idee anschließend auch mittels eines Ende-zu-Ende-Prototyps zu validieren.</p><p>Dies führt in der Regel zu langwierigen Diskussionen darüber, ob und wann mit der Validierung welcher Idee begonnen werden soll, ohne in dieser Zeit tatsächlich wertvolle Erkenntnisse über die Ideen selbst zu gewinnen. Dabei ermöglicht es eine geeignete Dateninfrastruktur insbesondere durch schnelle Ende-zu-Ende-Validierung und Experimente, Ideen schneller zu verwerfen. So muss über diese Ideen nicht ständig neu diskutiert werden und der Fokus kann leichter auf potentiell erfolgreiche Ideen gelegt werden. Die grundsätzliche Fähigkeit, Ideen mit schnellen Ende-zu-Ende-Prototypen und Experimenten zu validieren, ist insbesondere für die Entwicklung erfolgreicher ML-Systeme wichtig, da diese typischerweise aus vielen beweglichen Teilen bestehen und sich das System häufig über mehrere Teile des Unternehmens erstreckt.</p><p>Für ein Ende-zu-Ende-System müssen (Roh-)Daten geeignet aufbereitet werden, ein passendes ML-Modell entwickelt, trainiert und deployt werden und schließlich die durch das Modell generierten Erkenntnisse genau dort in einer brauchbaren Form zur Verfügung gestellt werden, wo diese schließlich einen Mehrwert liefern können. Darüber hinaus gilt es, noch weitere Herausforderungen wie Monitoring, Versionierung und Validierung mitzudenken.</p><h4>2. Der eigentliche Wert ist die Fähigkeit, ML-Algorithmen produktiv einzusetzen</h4><p>Typischerweise gibt es in Unternehmen eine Fülle von Ideen, wie Data Science und Machine Learning genutzt werden könnten, um Mehrwert zu erzeugen. Meistens sind auch bekannte ML-Algorithmen vorhanden, die bereits von anderen erfolgreich verwendet werden, um die gleiche oder eine ähnliche Idee umzusetzen. Es gibt daher oft berechtigten Grund zu der Annahme, dass dieser bekannte Algorithmus mindestens gut genug für eine erste Ver- sion des gewünschten ML-Systems ist.</p><p>Tatsächlich sind viele State-of-the-Art-Algorithmen und ML-Modelle heutzutage frei verfügbar:</p><ul><li>XGBoost, ein effizientes, flexibles und portables Gradient- Boosting-Framework, welches das Rückgrat für zahlreiche erfolgreiche ML-Systeme bildet.</li><li>ResNet, ein KI-Modell für moderne Bildklassifikation, ist in verschiedenen Open-Source-Frameworks wie Tensorflow und PyTorch implementiert und direkt einsatzbereit.</li><li>BERT, ein KI-Modell für Natural Language Processing, ebenfalls Open Source und in PyTorch and TensorFlow frei verfügbar.</li><li>Darüber hinaus stehen viele Algorithmen und Modelle auch „as a service“ bei Public-Cloud-Anbietern zur Verfügung und können dort direkt eingesetzt und oft sogar auf eigenen Daten feinjustiert werden.</li></ul><p>Zur Umsetzung der vorhandenen Ideen fehlt daher häufig nicht das Wissen über einen geeigneten Algorithmus, sondern es fehlt die Verfügbarkeit von Daten und insbesondere die Fähigkeit, den Algorithmus schmerzlos als ML-System Ende-zu-Ende in die Unternehmensinfrastruktur zu integrieren. Diese Punkte entscheiden am Ende daher regelmäßig darüber, ob mit einer Idee ein Mehrwert erzeugt werden kann – und nicht, ob ein passender Algorithmus gefunden wurde.</p><h2>Daten neu denken</h2><p>Wie schon erwähnt, ist das Überdenken von Daten und Dateninfrastruktur kein (rein) technisches Problem und beginnt typischerweise mit den Fragen:</p><ul><li>Welche Daten sind verfügbar?</li><li>Welche Qualität haben die Daten?</li><li>Wem gehören die Daten?</li><li>Wer kontrolliert den Zugriff auf die Daten?</li><li>Wer ist für die Daten und ihre Qualität verantwortlich?</li><li>Wie ist die Beziehung zwischen Datenproduzenten und Datenkonsumenten?</li><li>Wie kann diese (entscheidende) Beziehung verbessert werden?</li></ul><p>Ein Hauptantrieb für die Beantwortung dieser Fragen besteht darin, Wege zu finden, um die gesamte End-to-End-Zykluszeit zur Entwicklung von ML-Systemen zu reduzieren – indem beispielsweise die Verfügbarkeit, die Verarbeitungsgeschwindigkeit und die Qualität der Daten sukzessive verbessert werden.</p><p>In den meisten Fällen empfehlen wir, ausdrücklich nicht zu versuchen, alle Infrastrukturprobleme im Voraus zu lösen und danach mit Data-Science-Projekten zu starten. Im Gegenteil: Viele Schwierigkeiten werden erst offenbar, sobald die ersten ML-Systeme entwickelt werden, und sind vorab unbekannt. Daher ist es ist meist unmöglich, geeignete Lösungen für diese Schwierigkeiten zu finden, ohne konkrete Anwendungen zu implementieren und dabei zu lernen.</p><p>Letztlich gilt es, das Thema Daten in großem Maßstab zu überdenken, aber von Use Case zu Use Case im Kleinen umzusetzen. Dies ermöglicht zum einen die Validierung verschiedener technischer Ansätze. Zum anderen (und das ist wahrscheinlich wichtiger) erlaubt es uns, unsere Denkweise über die Bedeutung von Daten Schritt für Schritt zu ändern. In diesem Stadium bietet die Nutzung einer Cloud-Infrastruktur eine gute Möglichkeit, schnell und ohne große Vorlaufzeit und Anfangsinvestition zu arbeiten. Der Schwerpunkt liegt hier zunächst auf der Ende-zu-Ende-Umsetzung von Use Cases und insbesondere auf der Entwicklung produktionstauglicher ML-Systeme. Sobald tatsächlich mehrere Use Cases umgesetzt sind und einen Mehrwert schaffen, kann eine Vereinheitlichung der zugrunde liegenden Dateninfrastruktur möglicherweise einen zusätzlichen Mehrwert bieten.</p><h2>Data-Science-Projekte neu denken</h2><p>Data-Science-Projekte werden häufig mit einem Team aus Data Scientists gestartet. Während das auf den ersten Blick sinnvoll erscheint, verbergen sich in der Praxis hinter dem Begriff Data Scientist oft die unterschiedlichsten Skill Sets. Regelmäßig führt dies dazu, dass im Projektteam die Schwerpunkte der Fähigkeiten auf der Erstellung von ML-Modellen und deren Optimierung liegen, während die Themen Softwareentwicklung, Data Engineering und Produktentwicklung nur schwach vertreten sind.</p><p>In diesem Fall ist es häufig nicht zu erwarten, dass das Projektteam die verschiedenen Trade-offs, die im PoC eingegangen wurden und insbesondere deren Implikationen auf die Weiterentwicklung der Idee – etwa in Richtung eines Ende-zu-Ende Prototyps – transparent in Richtung „der Entscheidungsträger*innen“ kommunizieren kann. Zudem stellt die eigentliche Entwicklung eines solchen Prototyps das Team in vielen Fällen vor eine schwer lösbare Aufgabe, und es ist oft schlicht nicht zu erwarten, dass so ein schlanker Prototyp entsteht, der später schmerzlos iterativ angepasst werden kann.</p><p>Fehlt ausreichend Expertise für Produktentwicklung im Team, gelingt es in vielen Fällen zudem nicht, eine belastbare Produktvision für die Data-Science-Idee zu erarbeiten. Oft führt dies dazu, dass in Präsentationen vor „den Entscheidungsträger*innen“ der Fokus auf dem ML-Modell und den generellen Modell-Metriken liegt, wohingegen die Fragen „Was muss alles passieren, dass Geld verdient werden kann?“ und „Wie genau führen die Erkenntnisse des Modells nun eigentlich zu Mehrwert?“ zu stark in den Hintergrund rücken.</p><p>Wir empfehlen daher, Data-Science-Projekte in der Regel ausdrücklich nicht mit einem PoC und der Frage „Wie gut ist es möglich, mit KI XYZ zu erreichen?“ zu starten, sondern sich von Anfang an die Frage „Was ist notwendig, um mit der Idee XYZ einen Mehrwert zu generieren?“ zu stellen. Bei der Beantwortung spielt das KI-Modell natürlich eine Rolle, ist aber dann nur ein Faktor neben allen anderen Herausforderungen, die gemeistert werden müssen, um Mehrwert zu generieren. Entscheidend ist es dabei, das Team so aufzustellen, dass alle notwendigen Fähigkeiten vorhanden sind, um die ursprüngliche Idee wirklich Ende-zu-Ende validieren zu können. Ein Modell alleine ist hierfür in vielen Fällen nicht ausreichend.</p><h2>Zusammenfassung</h2><p>Während alle über KI sprechen, braucht es am Ende viel mehr als Data Science und ML-Algorithmen, um produktionstaugliche ML-Systeme zu entwickeln und Mehrwert zu schaffen. Insbesondere setzt nachhaltiger Erfolg mit ML-Systemen und KI mehr voraus als clevere Algorithmen – er erfordert eine Änderung der Einstellung zum Thema Daten, Dateninfrastruktur und einen ganzheitlichen Blick auf die mögliche Wertschöpfungskette des ML-Systems. Der wahre Wert für Unternehmen im Kontext von ML-Systemen liegt am Ende meistens nicht im Algorithmus, sondern in den zugrunde liegenden Daten und der Fähigkeit, diese Algorithmen schnell in Form eines ML-Systems auf der Dateninfrastruktur umzusetzen.</p><p>Ein Big-Bang-Ansatz, um die Dateninfrastruktur zu ändern oder damit zu beginnen, alle verfügbaren Daten an einem einzigen Ort zusammenzufassen, ist oft ein unnötig schwieriges Unterfangen. Meistens ist es erfolgversprechender, das Thema Daten erst einmal zu überdenken – von Use Case zu Use Case – und unterwegs Schritt für Schritt eine geeignete Dateninfrastruktur aufzubauen und insbesondere eine neue Datenkultur zu schaffen.</p>

KI, Daten und Infrastruktur – ML-Systeme schnell Ende-zu-Ende verproben und umsetzen

KI in der Praxis: Fehlerhafte Bauteile mit Rekognition auf AWS identifizieren

<p>Typically your favorite machine learning model doesn’t care whether or not your input dataset is professionally and technically correct. However, particularly for machine learning algorithms, the all-encompassing truth <em>garbage in, garbage out</em> holds true and hence it is strongly advised to validate datasets before feeding them into a machine learning algorithm.</p><p>Generally, validating datasets is a tedious task since we have to write a plethora of checks to ensure that the dataset contains all required columns and that the columns contain only expected values. Having written many dataset tests by hand, I was quite happy to stumble upon the Python library <a href="https://github.com/great-expectations/great_expectations" target="_blank">great_expectations</a> , which is a promising tool to validate datasets in a painless way.</p><p>In this blogpost, I want to introduce <code>great_expectations</code> and share some of my thoughts about why I think this tool is helpful in the toolset of every data person.</p><h2>The problem – why validate datasets?</h2><p>From a high-level point of view there are (at least) two kinds of problems occurring while engineering a dataset. First, there are more or less obvious technical errors such as missing rows or columns and wrong datatypes. Second, even when the actual data pipelines are solid and the datasets are put together in a technically correct way, there are often issues with degeneration of data over time. Here, too, we have obvious changes, e.g. additional categories in a categorical column. However, many changes in the data often go undetected. For example:</p><ul><li>the values of a binary column might be approximately evenly distributed between 0 and 1 at the beginning and the distibution could become skewed over time.</li><li>the mean value and standard deviation of sensor data emitted by a physical sensor could drift over time.</li></ul><p>Obvious changes in the data or mistakes while engineering the dataset typically lead to errors in the machine learning pipeline and hence are addressed as soon as they occur. The silent changes, however, are more subtle and they potentially impair the performance of the machine learning model visualized in the following picture.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/5htfswGESOu7OlK6XzbD"><img width="250" height="141" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/5htfswGESOu7OlK6XzbD" alt="validating datasets: data degeneration" /></a> </p><p>For this reason data monitoring and validation of datasets is crucial when operating machine learning systems.</p><p>In the following, we will look at a small example to introduce <code>great_expectations</code> as a tool for dataset validation.</p><h2>Small example</h2><p>In our example, we use the public domain <a href="https://www.kaggle.com/ajay1735/hmeq-data" target="_blank">hmeq-dataset</a>  from Kaggle. The context of the dataset is automation of the decision-making process for approval of lines of credit. However, in this blogpost we are not interested in the machine learning aspect of the problem. Instead, our goal is to use this dataset in order to show some ideas of the <code>great_expectations</code> library.</p><p>In this small example, we will take a short look at:</p><ul><li>Basic table expectations</li><li>Expectations for categorical data</li><li>Expectations for numeric data</li><li>Saving expectations and validating other datasets</li></ul><h3>Preliminaries</h3><p>The recommended way to follow the small example is to create a fresh <code>Python 3.8</code> environment and install <code>great_expectations</code> and <code>jupyter</code> via</p><div>

```bash
pip install great_expectations
pip install jupyter
```

</div><p>Then, we start a <code>jupyter-notebook</code> and import the library with</p><div>

```python
import great_expectations as ge
```

</div><p>Because <code>great_expectations</code> wraps the popular <a href="https://pandas.pydata.org" target="_blank">pandas</a>  Python library, we can use <code>pandas</code> functionality to import datasets. Hence, we may use</p><div>

```python
df = ge.read_csv('hmeq.csv')
```

</div><p>to read the dataset. In our example, we want to simulate a situation where we generate expectations for a dataset and then apply these expectations to validate, for example, a newer version of the dataset. For this reason, we execute</p><div>

```python
df = df.sample(frac=1).reset_index(drop=True)
split = int(len(df)/2)
df1 = df[:split]
df2 = df[split:]
```

</div><p>to shuffle the dataset and split it into two subsets. Now, we can create expectations using <code>df1</code> and validate the dataset <code>df2</code>.</p><h3>Basic table expectations</h3><p>We can generate hypotheses for the table with <code>great_expectations</code>. For example, we can use</p><div>

```python
min_table_length = 2500
max_table_length = 3500
df1.expect_table_row_count_to_be_between(min_table_length, max_table_length)
```

</div><p>if we have an idea how many rows our dataset should have. Typically, we require specific feature columns in our dataset for our machine learning algorithm. We can create expectations for columns to exist via</p><div>

```python
feature_columns = ['LOAN', 'VALUE', 'JOB', 'YOJ', 'CLNO', 'DEBTINC']
for col in feature_columns:
    df1.expect_column_to_exist(col)
```

</div><p>Table expectations provide simple sanity checks for the dataset. <code>great_expectations</code> manages all expectations in a <code>json</code> file. We can print all established expecations with</p><div>

```python
df1.get_expectation_suite()
```

</div><p>So far the <code>json</code> file should look something like this:</p><div>

```bash
{'data_asset_name': None,
 'expectation_suite_name': 'default',
 'meta': {'great_expectations.__version__': '0.8.7'},
 'expectations': [{'expectation_type': 'expect_table_row_count_to_be_between',
   'kwargs': {'min_value': 2500, 'max_value': 3500}},
  {'expectation_type': 'expect_column_to_exist', 'kwargs': {'column': 'LOAN'}},
  {'expectation_type': 'expect_column_to_exist',
   'kwargs': {'column': 'VALUE'}},
  {'expectation_type': 'expect_column_to_exist', 'kwargs': {'column': 'JOB'}},
  {'expectation_type': 'expect_column_to_exist', 'kwargs': {'column': 'YOJ'}},
  {'expectation_type': 'expect_column_to_exist', 'kwargs': {'column': 'CLNO'}},
  {'expectation_type': 'expect_column_to_exist',
   'kwargs': {'column': 'DEBTINC'}}],
 'data_asset_type': 'Dataset'}
```

</div><h3>Expectations for categorical data</h3><p>Besides checking the whole dataframe, we can also address specific columns. As an example of categorical data, we use the column <code>'JOB'</code>. First, we employ</p><div>

```python
df1.expect_column_values_to_be_of_type('JOB', 'object')
```

</div><p>to expect the correct dtype which typically is <code>'object'</code> in case of categorical data. Next, we can create an expectation for the expected values in the column with</p><div>

```python
expected_jobs = ['Other', 'ProfExe', 'Office', 'Mgr', 'Self', 'Sales']
df1.expect_column_values_to_be_in_set('JOB', expected_jobs)
```

</div><p>A very nice feature of <code>great_expectations</code> is the possibility to create expectations concerning the distribution of the column values. For this purpose we start by creating a categorical partition of the data.</p><div>

```python
expected_job_partition = ge.dataset.util.categorical_partition_data(df1.JOB)
```

</div><p>Then, we can use</p><div>

```python
df1.expect_column_chisquare_test_p_value_to_be_greater_than('JOB', expected_job_partition)
```

</div><p>to prepare a <a href="https://en.wikipedia.org/wiki/Chi-squared_test" target="_blank">Chi-squared test</a>  for comparing categorical distributions.</p><h3>Expectations for numeric data</h3><p>As an example of numeric data, we use the column <code>'LOAN'</code>. Again, we start with</p><div>

```python
df1.expect_column_values_to_be_of_type('LOAN', 'int64')
```

</div><p>to prepare a check for the correct dtype. In addition, we can use expectations such as</p><div>

```python
df1.expect_column_mean_to_be_between('LOAN', 10000, 20000)
df1.expect_column_max_to_be_between('LOAN', 50000, 100000)
df1.expect_column_min_to_be_between('LOAN', 1000, 5000)
```

</div><p>to ensure that min, max and mean of our data are in our expected ranges. Moreover, we can create a continuous partition of the data with</p><div>

```python
expected_loan_partition = ge.dataset.util.continuous_partition_data(df1.LOAN)
```

</div><p>and use</p><div>

```python
df1.expect_column_bootstrapped_ks_test_p_value_to_be_greater_than('LOAN', expected_loan_partition)
```

</div><p>to prepare a bootstrapped <a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test" target="_blank">Kolmogorov-Smirnov test</a>  for comparing continuous distributions.</p><h3>Save expectations and validate other datasets</h3><p>So far we have defined multiple expectations regarding the dataset <code>df1</code>. In practice, we would require additional expectations concerning other columns of our dataset. For the purpose of our (small) example we stop here. We can save the json file containing our expectations via</p><div>

```python
df1.save_expectation_suite('some_expectations.json')
```

</div><p>In our workflow, we can (and usually should) place the file <code>some_expectations.json</code> under version control. Now, we can use the expecations to validate other datasets.</p><div>

```python
df2.validate(expectation_suite='some_expectations.json', only_return_failures=True)
```

</div><p>In this case, we do not expect to encounter any errors because we randomly split the dataset into two subsets. However, we can see the validation come into play, for example, by dropping a column</p><div>

```python
df2_missing = df2.drop(columns=['LOAN'])
df2_missing.validate(expectation_suite='some_expectations.json', only_return_failures=True)
```

</div><p>or by setting a loan value which is too small</p><div>

```python
df2_min_low = df2.copy()
df2_min_low.at[4, 'LOAN'] = 10
df2_min_low['LOAN'] = df2_min_low['LOAN'].astype('int64')
df2_min_low.validate(expectation_suite='some_expectations.json', only_return_failures=True)
```

</div><h2>Conclusion</h2><p>In the example, we only covered a small subset of the available features of <code>great_expectations</code>. The tool offers more functionality such as</p><ul><li>more built-in expectations and even custom expecations</li><li>ways to integrate into data pipelines, e.g. with support for Spark</li><li>web-based data profiling and exploration</li><li>slack notification for failed validations</li></ul><p>which I have not used outside of small tests.</p><p>In my opinion, <code>great_expectations</code> appears to be a useful addition in the tool kit of each data scientist/engineer. It has a low barrier of entrance since it can basically be reduced to an additional <code>json</code> file living in the code repository, but it has the potential to significantly simplify validating datasets and, in particular, debugging data pipelines.</p><p>At the moment, I am not a great fan of the initialization via <code>great_expectations init</code> and the resulting folder structure in the project directory. However, I did not use <code>great_expectations</code> under real conditions and maybe there are advantages of this setup that I do not see at the moment.</p><p>Overall, <code>great_expectations</code> appears to integrate nicely in many machine learning pipelines and I cannot wait to extensively test the tool in future projects. If you have any experiences with <code>great_expectations</code>, feel free to share them in the comments.</p>

Great Expectations: Validating datasets in machine learning pipelines

Remote training with GitLab-CI and DVC

<p>While doing AI is sexy and cool, data infrastructure is typically not considered any of this. However, production-grade machine learning applications heavily rely on proper data infrastructure. Hence, in order to generate actual business value, solid data pipelines are required.</p><h2>A truth about data science projects</h2><p>Consider the following story: We start a data science project with a time-boxed proof of concept (PoC) having a skillful and enthusiastic project team that generates insights in fast iterations. After our time box ends, we achieved promising results and get the opportunity to present them in front of “the decision makers”. Luckily, we ace our presentation and it becomes more and more clear: “this idea is a winner!”. Everyone is feeling great.</p><p>However, under the hood, the promising results were achieved on a local machine with data dumps (CSV files) from multiple data sources carefully hand-crafted by an SQL expert with a special series of magical queries. Clearly, it is often very useful to work with data dumps for fast iterations, but clearly it is also a shortcut.</p><p>This shortcut becomes a problem when it is ignored. Even worse, when it is not properly communicated and understood by “the decision makers”. The trade-off for fast insights and results during the PoC was not considering data infrastructure and building proper data pipelines. Now, after everyone is hyped about the successful presentation and expects business value within “a few days”, we have to face the truth – we took a shortcut and before we can generate business value, we have to fix it.</p><h2>A heavy burden for data science projects</h2><p>Because it is possible to have short-term success as sketched in our fictional story, often times, the AI hype drives companies to start data science projects before evaluating and thinking about the current state of their data infrastructure. This can be useful in order to validate ideas. However, it becomes a problem if not all stakeholders are aware that the implications, i.e., building a production-grade machine learning application, are (depending on the data infrastructure), more probably than not, a challenging next step.</p><p>At worst, outdated and missing data infrastructure can lead to the bizarre situation where a data science project team is faced with the task to casually solve company-wide data infrastructure challenges. On the quest for generating business value after a successful PoC, the project team has to deal with organisational issues without much public awareness, and hence without much support – possibly resulting in frustration and failure to generate business value with the once so promising idea.</p><p>Metaphorically speaking, the team is driving a sports car (the shiny new machine learning model) on a forest path (the data infrastructure) and everyone around wonders why they do not run on full speed resulting in the impression that it is simply not possible to generate actual business value with data science projects.</p><h2>The attitude towards data infrastructure</h2><p>Historically, setting up a new data infrastructure is associated with great pain. This is because it is typically related to huge investments of time and money. Furthermore, there are various paradigms, e.g. data lakes (see <a href="https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/" target="_blank">here</a>  and <a href="https://jamesdixon.wordpress.com/2014/09/25/data-lakes-revisited/" target="_blank">here</a> ), <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html" target="_blank">data lakehousing</a>  and <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html#TheNextEnterpriseDataPlatformArchitecture" target="_blank">data meshes</a>  each with different sets of benefits and trade-offs to consider. However, improving the data infrastructure is expressly not solely a technical issue. In fact, what is even more important is changing the mindset about the significance of data in the company. This change is likely the single most crucial step for generating sustainable and growing business value with data science projects.</p><p>There are (at least) two reasons to rethink data infrastructure:</p><h3>Faster cycle time for validation</h3><p>Clearly, starting with a PoC as described above yields first insights as to whether or not an idea is promising. However, without proper data infrastructure, it is usually hard and costly to validate an idea end-to-end. This usually leads to lengthy discussions about if and when to start without actually starting to learn valuable lessons about the idea. Having proper data infrastructure allowing for fast end-to-end validation and experimentation enables one to dismiss unsuccessful ideas faster and makes it possible to single out the successful ones. The ability to validate ideas with fast end-to-end experiments is particularly crucial for building successful machine learning applications because there are many moving parts and the application typically affects various parts of the company.</p><h3>True value is the ability to utilize machine learning algorithms</h3><p>Typically, there are plenty of ideas on how to use data science to generate business value in a company. More often than not, there are also known machine learning algorithms already used by others to implement the same or similar ideas. Hence, there is legitimate reason to assume that this known algorithm supposedly would work at least well enough for a first version of the application. However, many times it is not the knowledge about the algorithm, but the availability of data and the ability to integrate the algorithm end-to-end into the company infrastructure in a painless way which decide if business value can be created.</p><h2>(Re-)thinking data</h2><p>Starting to (re-)think data is not only a technical issue; first it amounts to asking questions like</p><ul><li>What data is available?</li><li>What quality does the data have?</li><li>Who owns the data?</li><li>Who controls access to the data?</li><li>Who is responsible for the data and its quality?</li><li>What is the current relationship between producers and consumers of data?</li><li>How can we evolve and improve this crucial relationship?</li></ul><p>Here, the main driver for answering these questions is to find ways to reduce the overall end-to-end cycle time for the development of machine learning applications by improving, for example, the availability, the processing speed and the quality of data. In most cases, we do not recommend to actually solve all data infrastructure problems upfront and then start data science projects. In fact, many issues are typically unknown at this point and it is very challenging to find suitable solutions without implementing specific applications.</p><p>Hence, it is perfectly fine to start rethinking data on a large scale and act small on a use case by use case basis. For one thing, this allows validating different technical approaches. For another (this is probably more important), it allows changing the mindset about the significance of data step by step. The focus here is on implementing use cases end-to-end learning, in particular, to build production-grade machine learning applications. At this stage, using cloud infrastructure offers a good way to move quickly without much ramp-up. As soon as several use cases create business value, unifying the data infrastructure might yield additional value.</p><h2>Summary</h2><p>While everyone is talking about AI, in the end it takes much more than data science and machine learning algorithms to build production-grade machine learning applications and generate business value. Moreover, sustainable business value with machine learning and AI requires more than clever algorithms – it requires rethinking data.</p><p>In fact, many state-of-the-art algorithms are publicly available, e.g. the <a href="https://en.wikipedia.org/wiki/PageRank" target="_blank">Google search algorithm</a>  and even the <a href="https://patents.google.com/patent/US7058628B1/en" target="_blank">patent is expired</a> , the state-of-the-art model for natural language processing <a href="https://github.com/google-research/bert" target="_blank">BERT</a>  is open source and the network architecture for state-of-the-art image classification <a href="https://en.wikipedia.org/wiki/Residual_neural_network" target="_blank">ResNet</a>  is available and even implemented in various frameworks like Keras and Pytorch.</p><p>However, the major part of the business value is not the algorithm but the capability to employ the algorithm in a production-grade machine learning application, and hence the true value lies in the underlying data (often not open source) and the data infrastructure.</p><p>Here, a big bang attempt to change the data infrastructure or start by pouring all available data into a single place is more often than not an unnecessarily challenging approach. Instead, starting to rethink data – use case by use case – and building a suitable data infrastructure on the way appears to be much more promising.</p><p>If you are interested in some more ideas and our approach to data science projects, see the <a href="/wissens-hub/blog/wie-man-data-science-projekte-nicht-in-die-poc-sackgasse-manoevriert">blog post</a>  (German) and the free on-demand <a href="https://www.codecentric.de/wissens-hub/webinare/webinar-data-science-projekte">webinar</a>  (German).</p>

Thinking AI means re-thinking data

<p></p><p>This is the second article in our dish-o-tron series (a non-standard Deep Learning tutorial) in which we tackle one of the biggest problems in community kitchens: coming across someone else’s dirty dishes. We are facing this problem by building a state-of-the-art AI-system – the dish-o-tron.</p><p>If this is the first time you hear about the dish-o-tron and you are interested in the whole story, you might want to start with the <a href="https://blog.codecentric.de/dish-o-tron-no-more-dirty-dishes-ai">first part.</a> </p><p>In our conception, the dish-o-tron uses a computer vision AI model in order to detect dirty dishes, hence we require training data to produce this kind of AI-model. A brief look around reveals there is no suitable data available. This might come as a shock. However, this realization is a typical issue for many problem solvers tackling real-world problems with AI. Don’t get discouraged!</p><p>In this blogpost we start building the dish-o-tron hands-on by gathering an initial “good enough” data set for the next steps. Although we have already collected a dataset that we will share with you at some point, we need to point out that we will NOT share the link with you until you have collected your own data. In order to have the real dish-o-tron-experience, it is absolutely necessary that you gather training data yourself.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Ptc49Z3cSSWvrFc6feqP"><img width=640 src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/Ptc49Z3cSSWvrFc6feqP" alt="dish-o-tron" /></a> <br /> <i>image source: pixabay.com by Dieterich01</i></p><h2>Approach and reasoning</h2><p>The high-level idea is to just start tackling the problem with an end-to-end solution. For the first prototype it is reasonable to take some shortcuts. In many cases this is a very promising approach. We do not want to start by collecting data for a few months, then train “the best” AI-model for a few weeks and finally try to set up a dish-o-tron on a kitchen sink.</p><p>Instead, we’d like to put a first version of the dish-o-tron on a kitchen sink as fast as possible and iteratively improve the solution. In this way, we can (hopefully) decide with more certainty which parts actually need improvement by taking into account real-world feedback.</p><p>So in this first article, we gather and prepare an initial data set for the dish-o-tron in a hands-on way and in three steps:</p><ul><li>Making videos of clean and dirty kitchen sinks</li><li>Splitting the videos into images of clean and dirty kitchen sinks</li><li>Splitting the images into training, validation and test data sets</li></ul><p>As a starting point for your data collection we provide a Google Colab notebook to follow along. A Colab notebook is a service designed by Google that allows you to run code and train models in the cloud. And the best part: it is currently free. Think of a Colab notebook as a Linux Docker container that runs a web server that can execute Python code and even allows you to use a GPU for model training. You have a temporary file system in your container where you can download stuff and install libraries etc.</p><p>You can find the Colab notebook <a href="https://colab.research.google.com/drive/1tgXsHuG3yi-aoOL26y5DRwBovQC_PkhL" target="_blank">here.</a>  (You will need a Google account to be able to run the code. Save a copy of the notebook to your Google drive to persist your changes.)</p><h2>Videos of kitchen sinks</h2><p>As we have already mentioned several times (sorry but not sorry!), gathering data is a key step for many problem solvers to tackle actual real-world problems because these problems typically do not start with a polished Kaggle data set. For this reason, we strongly encourage you to leave the comfort of your desk chair and make videos of the kitchen sink in question.</p><p>Yes, at first glance, this is quite a hassle. However, gathering data on-site is a valuable learning experience because we obtain important information about the problem and its domain. In our case, this involvement with the domain, for instance, leads to questions like:</p><ul><li>What should the videos look like?</li><li>What data is useful and required for solving the problem?</li></ul><p>Thinking about such questions is important in order to tackle the actual problem at hand and not focus too narrowly, for example, on the AI component of the dish-o-tron.</p><p>A few further (possibly) helpful considerations about the videos:</p><ul><li>Take note of the future position of the dish-o-tron for the perspective</li><li>It might be useful to take several videos of clean and dirty sinks with little changes such as:<ul><li>switching the lights on/off</li><li>changing the position of the water tap</li><li>repositioning of unrelated objects around the sink</li></ul></li><li>It might help to move the camera slightly back and forth to add some variance</li><li>Dirty sinks come in many different configurations when making several videos with e.g. different plates/cups/cutlery, and changing their positions might be required.</li></ul><p>This is certainly not a complete list, feel free to point out additional considerations for the videos.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/d97f5igJQLujDf59Z0x7"><img loading="lazy" width="640" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/d97f5igJQLujDf59Z0x7" alt="kitchen sink" /></a> </p><p>Are you still sitting in your comfy desk chair? Did our passionate plea for the importance of gathering data not convince you? <p><b>You can do this! Gather data for your dish-o-tron!</b> <b>It’s worth it! Just grab your smartphone and follow these instructions:</b></p><ul><li>use a landscape perspective (please say NO to vertical videos)<br /> If you are not sure why you should say NO to vertical videos, please study this comprehensive explanation on  <a href="https://www.youtube.com/watch?v=f2picMQC-9E" target"_blank">YouTube</a>.</li><li>film from a top-down position (not from the front)</li><li>it’s allowed to have objects located next to the sink</li><li>the difference between clean and not_clean is only determined by whether or not there are dishes <b>IN</b> the sink</li><li>Don’t scroll down to read more. <b>Take your smartphone and go to the sink.</b></li></ul><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mHfedOJHSoeMEO1POg3Y"><img loading="lazy" width="640" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mHfedOJHSoeMEO1POg3Y" alt="kitchen sink" /></a> </p><p><span><b>PRIVACY WARNING: Make sure you do not record any personal things like photos or other people. Since you are recording video, also make sure you don’t record any conversations or other persons. Otherwise you won’t be able to share and talk about your great work later. Then you won’t become famous for building the best dish-o-tron in the world. As a result, you will not get the job as AI Lead at the self-driving car company and so on. So be careful, you have been warned.</b></span></p><p>Record 5 short videos (3-5 sec.) of a clean sink:</p><ul><li><ul><li>slowly move the camera slightly to get some different angles and reflections</li><li>for each video change some conditions e.g.<ul><li>switch light on/off</li><li>open the tap to make the sink wet</li><li>move the tap</li><li>… be creative – what else could happen?</li></ul></li></ul></li></ul><p><br /> (sample video for not_clean sink)</p><p>Record 5-8 short videos of a not_clean sink:</p><ul><li>put dishes/glasses/tools/pans whatever into the sink</li><li>for each video change something<ul><li>move the position of dishes</li><li>put more / remove</li><li>change light</li><li>…</li></ul></li></ul><p>That’s it. You have collected your very own first data to build the dish-o-tron. For the next steps this data will be enough. Will this data suffice to build a reliable AI product that works under every condition? Absolutely not! However, this data lays the foundation for building a running AI system and iteratively improving it.</p><p>Another source for additional data is your friends and colleagues. Just tell them about your journey to turn the community kitchen into a peaceful meeting ground. Ask them to provide additional videos of their dirty dishes for your collection. Believe it or not, this may be a nice door opener and starting point for interesting conversations with people to whom you didn’t talk for a while (and perhaps won’t for a while)!</p><h2>Labeling data</h2><p>Now go back to the <a href="https://colab.research.google.com/drive/1tgXsHuG3yi-aoOL26y5DRwBovQC_PkhL" target="_blank">Colab notebook</a>  and merge your data collection with the data that we provided. You can upload your additional videos into the Colab environment, for example via the UI in the panel on the left-hand side.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/OXJmNsZIT3KGwxs5vtYk"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/OXJmNsZIT3KGwxs5vtYk" alt="colab notebook" /></a> </p><p>Put them into <code>data/video_samples</code> and sort them into the right subfolders (first 5 into <code>clean</code> and the rest into <code>not_clean</code>). With this sorting step you have “labeled” the data. You told the dish-o-tron what a <code>clean</code> and a <code>not_clean</code> sink looks like. From your knowledge, dish-o-tron can learn everything!</p><p>That’s all the magic. You have put pictures into folders. Bravo.</p><p>OK, to be fair, labeling at scale is not that simple. Some datasets have millions of images. Some labels are not as easy to give as <em>clean</em> or <em>not_clean</em>. For example, a label could also be that you need to mark every pixel in an image where you see a road and seperate it from a wall. This might help a self-driving car to stay on track. Labels like this for millions of images can be very expensive – but also very valuable.</p><p><em>A short side note: Typically, AI systems benefit from large databases. Hence, we considered kick-starting a crowdsourcing campaign in order to gather a community DISH-O-TRON dataset. Potentially, this could improve all dish-o-trons around the world and crowdsourcing datasets would also be useful for various other kinds of problems. </em></p><p><em>In other IT communities, there are tools and platforms to collaboratively share and grow code. In some open-source software projects, hundreds or even thousands of collaborators are contributing to one big mission goal (often without getting anything back but good software). Unfortunately things like this do not yet exist to grow and collect datasets. But wouldn’t it be great to have a GitHub for datasets? To have a Kickstarter for data collection initiatives? To have hundreds of people around the world collecting data to feed the dish-o-tron? But maybe the data would just be too valuable to be shared – being the new oil or was it electricity? If you are interested in collaboratively building a large (dish-o-tron) dataset, please drop us a note.</em></p><h2>Splitting images in train, validation and test datasets</h2><p>A fundamental concept in training machine learning models is splitting the dataset into train, validation and test datasets. Because this concept is a comprehensive topic on its own, we only briefly discuss the intricacies of data splitting here. To get a better understanding, we strongly recommend familiarizing yourself with this topic further. A possible starting point is the articles <a href="https://www.fast.ai/2017/11/13/validation-sets/" target="_blank">here </a>  and <a href="https://mlbook.explained.ai/bulldozer-testing.html" target="_blank">here.</a>  (For our German speaking readers: You could also watch our introduction to machine learning video from our AI bootcamp <a href="https://youtu.be/iX9r8wvjKdo?t=1167" target="_blank">here</a> .)</p><p><em>Another side note: <a href="https://www.fast.ai/" target="_blank">fast.ai</a>  is a really great starting point if you want to learn more about machine learning. Big kudos to Jeremy Howard, his team and the fast.ai community. You have been a great inspiration to us as well. We have watched your lessons. We love your practical way of teaching. Implement something and learn by doing! It’s not necessary and also not possible to understand all details of Deep Learning before you build something. Like this you will never build something. That’s how the dish-o-tron was born, by the way. </em></p><p>Choosing a validation dataset and test dataset in order to evaluate the model more or less defines the rules of the game. Hence, it is crucial to understand the implications of the chosen splitting approach. For example, in our case, the images originate from videos, and hence are not completely unrelated in this sense. Two chronologically close frames of a video might not be very different potentially resulting in two very similar images in the train and test dataset.</p><p>In the future it could be useful to have a test set for the dish-o-tron containing only images from kitchen sinks that are not present in the training and validation dataset. Many times it is not clear at the beginning what the test set should look like and creating a reliable test set is an art in itself. In many cases, we have to iterate and improve it over time.</p><p>A rule of thumb is that the test set should represent the actual real-life situation as well as possible. Therefore, there is a good argument in favour of putting data from the same kitchen sink into the train, validation and test set if the model for the dish-o-tron is only used for one particular sink.</p><p><strong>Attention:</strong> the Colab notebook only provides temporary storage and all data will be deleted if the notebook is closed. Hence, to persist the data, you have to download it or store it, e.g. in your Google Drive storage.</p><h2>Conclusion</h2><p>In this article we tackled a sub-problem that appears to be tedious at first glance. However, gathering data, working and preparing data, understanding the data and its origins are fundamental tasks for problem solvers to understand the big picture of the problem at hand.</p><p>We hope that we were able to motivate you to actually get up from your desk and take videos of kitchen sinks in various configurations and work with the data. This is an important step to build the dish-o-tron and get the real problem solver experience.</p><p>In the next article we will use that data to train our first model. We will demonstrate this by using a service like Google AutoML as well as an easy-to-use framework like fast.ai. If you want to get in contact with us and maybe ask for the whole DIRTY-DISHES-DATASET, please answer to this <a href="https://twitter.com/MoserOli/status/1306864868490870784" target="_blank">tweet. </a> </p><p>Continue with the <a href="https://blog.codecentric.de/dish-o-tron-train-that-vision-model" target="_blank"> third part</a>  of our series where we train the vision model.</p>

DISH-O-TRON – Gather that DATA you must!

<p></p><p>With this article we continue our endeavor of building dish-o-tron – an AI system designed to prevent the sudden appearance of dirty dishes in the community kitchen sink, and hence turning the community kitchen into a place of peace and harmony.</p><p>This is part 3 of the dish-o-tron series, you may want to start with the <a href="/wissens-hub/blog/dish-o-tron-no-more-dirty-dishes-ai">first part</a>  where we introduce the idea and the concept behind dish-o-tron and the <a href="/wissens-hub/blog/dish-o-tron-gather-that-data-you-must">second part</a>  where we collect the initial data set.</p><p>In this article, we use the data gathered in the previous part to build “the heart” (or – perhaps better – “the brains”) of dish-o-tron empowering it to detect dirty dishes. In concrete terms, we train a machine learning model which is capable of classifying images of sinks into clean (no dirty dishes) and not_clean (dirty dishes) using the fast.ai library and AutoML from Google Cloud.</p><p>If this is the point where you think to yourself “oops, I did not gather any data” – we warned you several times. It is absolutely necessary that you gather training data yourself to have the real dish-o-tron-experience. We strongly encourage you to revisit the previous article and gather your own data and, in particular, don’t download our pre-prepared <a href="#">dataset</a> .</p><p></p><p>If this is the point where you think to yourself “Yay, I did gather my own data”: congratulations, you may now continue your journey and indulge in one of the most favourite occupations of every deep learner: watching an AI-model during training.</p><p>If you are a developer maybe you know watching your program compile or watching your CI/CD pipeline running tests. But watching an AI model train is something special. And if you watch harder (but not too hard) you might even influence the accuracy of the outcoming model! Depending on the architecture it might be necessary to watch deeper instead of harder – you will find out with further practice.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/JNowHiDoSdaBTGA0wtlf"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/JNowHiDoSdaBTGA0wtlf" alt="dish-o-tron: codes compiling, model is training meme" width="250" height="126" /></a> <p>Comics taken from <a href="https://xkcd.com/303/">1</a>  and <a href="https://www.reddit.com/r/ProgrammerHumor/comments/9cu51a/shamelessly_stolen_from_xkcd_credit_where_is_due/">2</a>  kudos to XKCD</p></div><p>We start with a short excursion about the requirements of dish-o-tron.</p><h2>Excursion: Revisit some requirements of dish-o-tron</h2><p>The dish-o-tron needs to be able to set off an alarm in case of a coworker violating the general rules of using the community kitchen. In most kitchens there are rules are like:</p><ul><li>DO NOT PUT DIRTY DISHES IN THE SINK!</li><li>Please, respect rule number 1 !!!1!eleven!</li><li>If the dishwasher is running, take your stuff and leave it at your desk until the dishwasher has finished.</li><li>EVERYBODY can empty the dishwasher.</li><li>NO EXCUSES. DO NOT PUT DIRTY DISHES IN THE SINK. NEVER.</li></ul><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/h2VWosxkSv21T3UaQPK5"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/h2VWosxkSv21T3UaQPK5" alt="dish-o-tron: community kitchen rules" width="247" height="300" /></a> <p>Exemplary community kitchen rules</p></div><p>In many kitchens these rules are manifested on various posters, stickers and even laminated printouts! Some even colorize words (OMG!) to emphasize that really everybody should take care of this. But we all know it. We are rebels. While reading these signs one always thinks: One day, when nobody sees me, I will just put my cup in the sink and run!</p><p>So far, we are not sure if this is only a German thing and thus if you have rules like this in your community kitchen please share a pic by answering to this <a href="https://twitter.com/MoserOli/status/1306864868490870784" target="_blank">tweet. </a> .</p><p>Because there is nothing we can do about this – we have to find another solution. The next reasonable step obviously is: Permanent control and punishment. That’s where dish-o-tron enters the arena. Inspired by the DEFCON levels of the United States Armed Forces we therefore propose the DISHCON levels (see <a href="https://en.wikipedia.org/wiki/DEFCON">this</a>  wikipedia article for reference.)</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qofjtBnRB2W51gb2QP6z"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qofjtBnRB2W51gb2QP6z" alt="dish-o-tron: table with dishcon levels" width="250" height="68" /></a> <p>The dishcon levels.</p></div><p>Since we are peace-loving problem solvers the escalations for DISHCON 1 and 2 WILL NEVER be implemented. Also privacy is important to us, so we will not record or save any images. We will not transfer any footage to the cloud. Dish-o-tron sees, maybe beeps, and then it just forgets.</p><h2>Approach and Reasoning</h2><p>Until only recently training a machine learning model for image classification would have required special knowledge in Data Science, however, current progress and development in particular in the environment of public cloud providers significantly simplified this task for problem solvers looking for rapid end-to-end progress.</p><p>This low barrier of entry into AI systems allows us to rely on existing libraries such as fast.ai and services like AutoML from Google Cloud to obtain a reasonable state-of-the-art vision model for our classification task. In this way we can build the first functioning prototype and focus on solving the actual problem at hand. At a later stage it might be useful to revisit the model training, however, the best model is useless as long as it is not integrated.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/nCond4pDT4QKjh3Dhmb1"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/nCond4pDT4QKjh3Dhmb1" alt="sidetracked meme" width="700" height="235" /></a> </p><p>For many people dealing with AI and building neural networks from scratch is a lot of fun. However, be honest with yourself! There is close to zero chance that you will create something that will come close to existing solutions. In fact, you will spend lots of time for a worse outcome. It is essential that you focus! Don’t get sidetracked! You are a problem solver. Your goal is to solve an actual real-world problem. The AI model is merely a tool for you to bring peace and harmony to your community kitchen.</p><p>In the following, we pursue two options to obtain a vision model in just a few steps:</p><ul><li>We utilize the fast.ai library</li><li>We use AutoML in the Google Cloud</li></ul><p><em>Short sidenote: Yes, it might be useful to revisit the vision model at some point. At this stage of the project it is helpful to think about this point in time in terms of “as soon as 80% of all community kitchen sinks are equipped with a dish-o-tron”.</em></p><h2>fast.ai</h2><p>fast.ai is per se a great starting point if you want to start with deep learning and machine learning. With the mission of “Making neural nets uncool again” it provides a competitive high-level python library allowing for rapid progress while building an AI system.</p><p>The fast.ai library allows you to train state-of-the-art vision models in a few lines of code. To get started you use the following colab-notebook:<br /> <a href="https://colab.research.google.com/drive/136Ios1VaMjQ-fJDsUi7umq41aX7NVmaT?usp=sharing" target="_blank">Colab notebook</a> </p><p>When finishing this notebook you will end up with a fast.ai model which is basically a pytorch model. This model can also be exported and used outside of the colab-notebook environment. However, so far we struggled a lot to deploy fast.ai models on edge devices and in particular on a Google Coral device. Somehow we did not find a painless way to do so. Feel free to investigate on your own and we are very happy if you reach out to us if you find a nice way.</p><h2>AutoML</h2><p>AutoML is a Machine Learning Service from Google Cloud which allows you to automate the training of your own custom vision models. It comes with a graphical interface and the option to, e.g., export models to edge devices such as the Google Coral device. The only thing you have to provide are labeled images and money. Yes, that’s basically it: you trade money for AI-expertise and speed. For training a model with ~10.000 labeled images we expect costs of ~25 $.</p><p>Does this mean AutoML is always the right solution? Not at all! But it is a nice tool to have if you are looking for rapid end-to-end progress. This is particularly the case if the goal is to validate ideas. Here, learning slowly and struggling to make any real end-to-end progress with an idea in favour of saving a few bucks on your cloud bill is often the worst choice.</p><p>Obtaining an AutoML vision model requires four simple steps:</p><ol><li>A tiny bit more data preparation and uploading the data</li><li>Creating the Dataset in AutoML</li><li>Training a readily available computer vision model in AutoML</li><li>Export the model (in a suitable format for the Coral device)</li></ol><p>In order to follow along you require access to the Google Cloud and a Google Cloud project ideally with <code>project-owner</code> access privileges.</p><p><em><strong>ATTENTION:</strong> Not everything we do is covered by the free-tier and hence, some charges may apply.</em></p><h3>1. Data preparation</h3><p>Before we can use AutoML to train a vision model, we have to upload our data to Google Cloud and also prepare a CSV file containing meta information about the data such as, e.g., labels of the images. This is a necessary evil before we can finally lean back and throw some money at Google to do the rest of the work.</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mGbRQvrmS02hhbTGaTft"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mGbRQvrmS02hhbTGaTft" alt="shut up and take my money meme" width="250" height="141" /></a> </p><p><a href="https://colab.research.google.com/drive/1Fow9mUGSICF7gCdoyACJvI3n3TjZlfib?usp=sharing" target="_blank">This Colab notebook</a>  should help you to take the final hurdle. Here, we provide a possible way to:</p><ul><li>Upload our data into a Storage Bucket in Google Cloud</li><li>Generate the necessary metadata CSV-file for AutoML</li></ul><p>Finally, we are in a position to use AutoML.</p><h3>2. Creating the dataset in AutoML</h3><p>The starting point for using AutoML is creating a dataset. Because we already uploaded the data into a GCS bucket and prepared the CSV metadata file, we can create the dataset with a few clicks in the UI. After triggering the upload the import will take some time. This is your chance to ponder about life and do some meditation. You could also watch some cat videos – if this is your thing – or just grab a cup of coffee. While you are in the kitchen there might be an opportunity to collect another dirty dishes video. Don’t get mad – you already made fantastic progress on your journey to build dish-o-tron.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/1zsudeBESL6gFqlv3ci6"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/1zsudeBESL6gFqlv3ci6" alt="creating a data set in automl step 1" width="250" height="114" /></a> <p>Creating a data set in AutoML.</p></div><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2JNWp6NJTcahGYuotHxZ"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2JNWp6NJTcahGYuotHxZ" alt="creating a dataset in automl part 2" width="250" height="151" /></a> <p>Using the prepared csv file.</p></div><p>As soon as the import is finished, we can inspect the dataset in AutoML. It is useful to make a few sanity checks at this point to ensure that the data is uploaded correctly.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/dURFuIRUGrGwkKmAoAtw"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/dURFuIRUGrGwkKmAoAtw" alt="Inspecting a dataset in AutoML" width="250" height="113" /></a> <p>Inspecting a dataset in AutoML</p></div><h3>3. Training the vision model</h3><p>And now, finally, it is going to happen. We can start training the model with a few clicks in the UI. Because we plan on deploying the model on a Coral device, we choose the option “Edge”. For simplicity we select “optimize for best trade-off between latency and accuracy” and set (depending on the number of images) a suitable amount of node hours.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/S6egV82hRWerd4KoFEcY"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/S6egV82hRWerd4KoFEcY" alt="table with node hours depending on number of images" width="250" height="88" /></a> <p>Suggested node hours depending on number of images in the dataset.</p></div><p>Please be aware that for each unit of time, Google Cloud uses 8 nodes in parallel, where each node is equivalent to a n1-standard-8 machine with an attached NVIDIA® Tesla® V100 GPU. Hence 8 node hours are approximately 1 “wall clock” hour. It is advisable to use the early stopping feature to ensure that training stops when further accuracy improvement is not possible. In the end, you pay only for the compute hours that are actually used.</p><p>Now push the final button.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tkDuIuA1TNeZlKpH5ZdN"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/tkDuIuA1TNeZlKpH5ZdN" alt="start model training in automl" width="250" height="240" /></a> <p>Start model training in AutoML</p></div><p>You did it! You are now a real Deep Learner! Feel free to relax for a few hours and check at irregular intervals if the training is finished. This is your time to take a break without feeling bad about it. That is what being a Deep Learner is all about.</p><p>Training a model is a magical experience. Don’t forget to check on your model and observe it during the training every once in a while: Rumour has it that observing the training procedure will change the outcome of the experiment. There are even stories that the intensity of the observing influences the accuracy of the model.</p><p>When the training is complete or at the latest, when you are back at your desk and observe that the training is complete, it is time for a few sanity checks of the model. Again this is possible with built-in validations of AutoML. If the accuracy is below 95% there is a strong reason to believe that something went wrong with the data or the data preparation.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/jBmDFNdRSrKDoI76UjuJ"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/jBmDFNdRSrKDoI76UjuJ" alt="Model evaluation in AutoML" width="250" height="119" /></a> <p>Evaluating a trained model in AutoML</p></div><p>If everything looks fine, we export the model for coral devices.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mj7LYAgQR6Cwk1pFItMA"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/mj7LYAgQR6Cwk1pFItMA" alt="exporting a trained model in AutoML" width="250" height="207" /></a> <p>Exporting the trained model in AutoML</p></div><p>That’s it! We have our first vision model for our dish-o-tron. Peace and harmony for your community kitchen were rarely as tangible as at this point in time.</p><h2>Conclusion</h2><p>Finishing this part of the tutorial is an important step for you and your future career as a professional problem solver. Frankly, that’s one (very) small step for Deep Learning, one giant leap for you – but that is okay. Be proud of yourself! This is how successful real-world problem solvers tackle AI tasks for the first iterations.</p><p>Okay, let’s make this more official: you have earned the AI TRAINING WATCHER badge (silver level)</p><p><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/G1Aqu5RLivGOERab3gvo"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/G1Aqu5RLivGOERab3gvo" alt="AI-Model watcher badge" width="250" height="135" /></a> </p><p>Don’t be shy, you earned it! Feel free to print it out and proudly wear it however you enjoy!</p><p>In the next article, we will build the first physical version of DISH-O-TRON which can (and should) be put into use at a real community kitchen sink. Stay tuned!</p>

DISH-O-TRON – Train that vision model!

<p><em>Sadly, to tell you the truth, doing dishes is still a thing. However, so far most of our readers still like our non-standard Deep Learning tutorial.</em></p><p>Typically, AI is demonstrated as solving various toy problems. AI plays chess and Go, AI plays video games, AI makes people dance. It is time to stop this madness and finally apply AI in a meaningful way. Therefore, we proudly present the dish-o-tron. The dish-o-tron is an AI system designed to solve an actual real-world problem impacting millions of people around the world every day: facing dirty dishes in the community kitchen sink.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/pcyYzx0XQNaMmoRiwcBn"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/pcyYzx0XQNaMmoRiwcBn" alt="dirty dishes in the community kitchen sink" width="250" height="107" /></a> <p>dirty dishes in the community kitchen sink – a real-world problem</p></div><p>Reading this blog series will equip you with the ultimate power to solve this long-lasting problem in your community kitchen once and for all by using state-of-the-art AI technology.</p><h2>The dish-o-tron</h2><p>At first glance, the dish-o-tron is an inconspicuous, well-positioned webcam in the kitchen observing the shared kitchen sink. In its natural state the dish-o-tron is just happy and enjoys life. The dish-o-tron doesn’t care whether you prefer tea or coffee and it likes all kinds of kitchen talk. However, there is one single thing that the dish-o-tron absolutely hates: watching someone put dirty dishes in the community sink.</p><p></p><p>Detecting dirty dishes in the sink enrages the peace-loving dish-o-tron so much that it starts beeping. The only way to return it to its natural peaceful state and thus stopping the noise is to admit one’s mistake and remove all dirty dishes from the community sink, leaving it neat and clean again.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/KshoGpISnyD4NikRB1A8"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/KshoGpISnyD4NikRB1A8" alt="respect privacy in the kitchen" width="250" height="166" /></a> <p>privacy in the kitchen has to be respected.</p></div><p>Building the dish-o-tron requires three high-level steps:</p><ul><li>Gathering and preparing data</li><li>Training an AI model</li><li>Deployment of the model</li></ul><p>In the following, we will discuss these steps further.</p><h2>Gathering and preparing data</h2><p>Trying to solve real-world problems with AI often starts with the realisation that there is little or even no data available. This issue prevents many problem solvers from actually solving the problem. “If only data collection had started years ago!”, they say, “then we could now actually solve the problem”. While this is a reasonable thought, it simply doesn’t help.</p><p>Consoling users currently facing a problem by saying that it is necessary to gather lots of data for quite some time before we can start building a solution is at least challenging. Typically a more promising approach is to build a system addressing the problem which is able to improve over time.</p><p>In this way, we will not solve the problem completely in the first step; however, we will tackle the problem right away and put ourselves in a position to iteratively adjust the solution to match the requirements which also become more and more clear while working on the problem.</p><p>Since our problem is unique in a sense that there is no Kaggle dataset readily available, we start our journey to building the dish-o-tron by doing our best to collect a suitable dataset for a first working system. Here, we will make videos of various kitchen sinks clean and not clean and split them up into a first labeled dataset.</p><p></p><p>In this way, we started collecting the DIRTY-DISHES-DATASET with thousands of pictures that we will share with you in the next article.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/JuNFMzUpTqRtwMfQmcOS"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/JuNFMzUpTqRtwMfQmcOS" alt="sample images from the dirty-dishes dataset" width="250" height="151" /></a> <p>sample images from the dirty-dishes dataset</p></div><h2>Training an AI model</h2><p>Not so long ago, training an AI model was tedious and required expert knowledge. In many cases this is still true today. Depending on the problem, we have to figure out a suitable model architecture and feature engineering and this requires some experimentation before we can train a suitable AI model. This is another issue which prevents problem solvers from building a solution tackling the whole problem even if data is available.</p><p>Fortunately, image classification is one of the best understood use cases in AI. There are lots of established best practices regarding model architectures and training of models. Among others this led to two things:</p><ul><li>High-level software libraries such as fast.ai which abstract away lots of the nitty-gritty details of image classification, providing a black-box kind of approach where state-of-the-art practises are simply utilised without burdening the user with the details.</li><li>Machine Learning as a service offerings from various public cloud providers such as automl and rekognition allowing training of image classification models on custom data in a few simple steps.</li></ul><p>Both approaches will typically not lead to the absolutely best solution. However, most of the time this is not necessary and ‘good enough’ will be just fine and a nice trade-off between time & money spent vs. result. For our first version of the dish-o-tron, we will employ the <a href="https://cloud.google.com/automl" target="_blank">AutoML Service</a>  from Google Cloud to train a first model.</p><p>We can use various tools to inspect the model and try to explain if the black box learns what we expect.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/VpD61VqpQnyJhXGaIBAF"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/VpD61VqpQnyJhXGaIBAF" alt="visualizing what the dish-o-tron model has actually learned" width="250" height="191" /></a> <p>Visualizing what the model has actually learned.</p></div><p>The training of the AI model with AutoML and its technical details will be discussed in a follow-up blog post.</p><h2>Deployment of the model</h2><p>Having an AI model generally will not solve an actual real-life problem. For a viable solution, the AI model has to be integrated into a suitable context. Many times, this is the key step to generating any value at all. Nevertheless, this step is often postponed to the distant future after “collecting high quality data” and “building the best AI model”. This is, more often than not, a mistake because integrating the model into its context poses various challenges on its own. Hence, it should not be ignored and instead tackled early in order to learn and identify the associated challenges.</p><p>While building the dish-o-tron, we tried multiple options to run the model. We deployed it on a Pi Zero which is a really small and cheap device that can be glued anywhere with a small powerbank. But it is rather slow. We ran the model in the browser using our notebook’s webcam with TensorFlow.js. We used the Google AIY Kit, which is much faster than the Pi Zero and also comes with a beeper and blinking lights (but it is quite old and deploying state-of-the-art models is hacky). Finally, we used the Google Coral device, which is made for this kind of workload and well-integrated into Google AutoML but comes with a price tag.</p><p>The community kitchen is a special place. It’s a place where rumors are born, where gossip is produced and where you can openly chat about the most secret secrets of your company! That’s why dish-o-tron is living on the edge. Edge devices enable you to run audio and video analytics AND respect the privacy of your community kitchen. No image is transferred to the cloud. Nothing is saved. Dish-o-tron sees and forgets.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ifYte6hjS9qkGzscSWIv"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/ifYte6hjS9qkGzscSWIv" alt="various edge devices" width="250" height="85" /></a> <p>Various edge devices</p></div><p>Moreover, the hardware we consider and buy in order to actually build the dish-o-tron will establish basic conditions for our solution space. In other words, we have to mind that it is possible to painlessly deploy the AI model on our preferred edge device. For the first version of the dish-o-tron, we decided to use a Google AIY kit (see video below). For the next version, we chose a Google Coral edge device, which allows us to run advanced computer vision tasks on a Raspberry-size mini computer. Fortunately, AutoML allows us to export models in a viable format.</p><div><a href="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/prd2PYdSwOt0zBb9m6Ph"><img loading="lazy" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/prd2PYdSwOt0zBb9m6Ph" alt="Google coral device" width="250" height="131" /></a> <p>Google coral device</p></div><p>The construction of the dish-o-tron including the deployment of the model on the Coral device and its technical details will be discussed in an upcoming blog post.</p><h2>Conclusion</h2><p>AI research has brought us new technology that can solve problems that couldn’t be solved before. Have you read the book <em>AI superpowers</em> by Kai-Fu Lee? He says that you don’t need to be one of the best AI researchers any more to apply AI and find new business opportunities. You need to collect (lots of) data and can “just” use existing algorithms, services and open source frameworks. Well, in our opinion building AI solutions is not easy – but it is indeed getting easier and easier every day.</p><p>See the first prototype running on the google AIY kit here (mind the green/red LED at the box):</p><p></p><p>Follow this blog series if you want to know how to build and run such a model on an edge device yourself. Building the dish-o-tron will fundamentally change the way you experience the community kitchen. Instead of being a place of constant anger and hostility, the community kitchen will become a peaceful meeting ground for sharing ideas and connecting with co-workers.</p><p>In the upcoming blog posts, we will guide you through the process of building your own dish-o-tron for your community kitchen sink. Hence, we will tackle a real-world problem and playfully learn how to build and improve an AI system from scratch. Stay tuned!</p><p>Continue with the <a href="/wissens-hub/blog/dish-o-tron-gather-that-data-you-must">the second part of our series</a>  where we start with gathering data.</p>

DISH-O-TRON – No more dirty dishes thanks to AI

<p>Im April 2018 haben wir bei codecentric eine AI (Artificial Intelligence bzw. KI = künstliche Intelligenz) Initiative gestartet. Unter dem Projektnamen „<a href="/leistungen/data-science-und-kuenstliche-intelligenz">codecentric.AI</a> “ werden wir versuchen zu zeigen, was heute mit künstlicher Intelligenz möglich ist (und was nicht). Man hört und liest überall von neuronalen Netzen, NLP, Computer Vision, CNNs, RNNs, LSTMs, AI, KI, Deep Learning und vielen weiteren Buzzwords.<br /> Wir werden die aktuellsten Technologien ausprobieren und darüber berichten, Tutorials veröffentlichen, interessante Paper vorstellen und in Demos zeigen, welche neuen Möglichkeiten sich durch Deep Learning & Co. KG ergeben.</p><p><img width="800" height="392" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/OvtVGy9ARwu6QbUB1Rnt" alt="Banner künstliche Intelligenz" /></p><p>Um diese Themen richtig darstellen zu können, haben wir einen speziellen <a href="https://www.youtube.com/channel/UC_r-M-0JQZUnOG2gJSL9pfg" target="_blank">YouTube-Kanal</a>  eingerichtet. Dort werden wir nur Videos zum Thema „AI“ einstellen. Hier siehst du ein erstes Beispiel zum Thema Object Detection. Wir fliegen mit einer Drohne durch unser Headquarters in Solingen und lassen Detectron (ein neuronales Netz von Facebook) Objekte detektieren:</p><div><div><div><div> <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/8LnnvnWTQir5XNkscjGQ" alt="YouTube" /><div><p>Mit dem Laden des Videos akzeptieren Sie die Datenschutzerklärung von YouTube.<br /><a href="https://policies.google.com/privacy" target="_blank">Mehr erfahren</a> <br /></p><p><a href="#">Video laden</a> </p><p> <small>YouTube immer entsperren</small></p></div></div></div><div></div></div></div><p>Wenn dir das Video gefallen hat und/oder wenn dich das Themengebiet interessiert, kannst du unseren <a href="https://www.youtube.com/channel/UC_r-M-0JQZUnOG2gJSL9pfg" target="_blank">Kanal</a>  abonnieren, um kein weiteres Video zu verpassen. Gerne kannst du auch in einem Kommentar schreiben, welche Themen dich besonders interessieren würden.</p><h2>Aber gibt es nicht schon genug AI Buzzwords?</h2><p>Bei codecentric arbeiten erfahrene Data Scientists und wir setzen Machine Learning und neuronale Netze bereits in diversen Projekten ein. Dennoch sind wir der Meinung, dass in Deutschland noch zu wenig Know-How in diesen Bereichen vorhanden ist.</p><p>Große Firmen wie Amazon, Google und Facebook investieren unvorstellbare Summen in AI/KI und haben bereits einen scheinbar uneinholbaren Vorsprung erarbeitet. Einige Forscher sind der Meinung, dass der Impact von künstlicher Intelligenz in naher Zukunft größer sein wird als der des Internets. Wir sind zwar (mindestens) noch Jahrzehnte von einer AI entfernt, die annähernd so viele Fähigkeiten wie ein Mensch kombinieren kann – dafür müsste allein schon die existierende Hardware noch um Größenordnungen schneller und besser werden. Aber die existierende KI-Technologie wird weitere Branchen auf den Kopf stellen, die bisher noch als „nicht automatisierbar“ gelten – da ist nicht mehr die Frage, ob es passieren wird sondern nur noch wann.</p><p>Es gibt viele Leute, die sich vor Jahren schon einmal mit neuronalen Netzen beschäftigt haben. „Damals“ schien es so, als sei die Technik noch nicht bereit, um damit sinnvolle Produkte bauen zu können. Viele sehen daher das Thema etwas voreingenommen und betrachten die aktuelle Entwicklung nur als Buzzword-Bingo. Auch mir ging es im Bereich NLP so. Ich habe lange gebraucht, um meine Vorurteile zu überwinden und zu bemerken, dass mein „Google-Navi“ inzwischen sehr gut versteht, wo ich hinfahren möchte und es absolut nicht mehr notwendig ist, die Zieladresse fummelig auf dem Handy einzutippen.</p><p>In den letzten Jahren ist sehr viel passiert. Gerade im Bereich Computer Vision wurden riesige Fortschritte gemacht. In der Forschung kommen so viele Papers heraus, die den State-of-the-Art immer weiter nach vorne treiben, dass man diese gar nicht mehr alle lesen kann. In der Computer Vision wurden Good Practices erarbeitet, die es ermöglichen mit weniger Daten viel bessere Ergebnisse zu erzielen (z.B. pretrained networks, transfer learning, fine tuning etc.). Diese Good Practices haben zu einer enormen Verbesserung geführt und auch dazu, dass „jedermann“ mit breit verfügbaren Daten in das Gebiet einsteigen kann. In anderen Disziplinen (z.B. NLP) werden viele Good Practices aber noch gar nicht oder kaum umgesetzt – das heisst, dass auch dort weiterhin mit grossen Fortschritten zu rechnen ist.</p><p>Ich bin der Meinung, dass es sich lohnt, das Thema noch einmal unvoreingenommen zu betrachten. Ausserdem denke ich, dass sich deutsche und europäische Konzerne viel mehr mit den Themen beschäftigen sollten und Experten-Teams und Know-How aufbauen müssen, um nicht den Anschluss zu verlieren.</p><p>Wenn ihr also auf dem Laufenden bleiben wollt, was in der Welt der Künstlichen Intelligenz so möglich ist und euch vielleicht auch selber mal an der ein oder anderen neuen Technik versuchen wollt, dann abonniert unseren <a href="https://www.youtube.com/channel/UC_r-M-0JQZUnOG2gJSL9pfg" target="_blank">codecentric.AI YouTube-Kanal</a> !</p>

Künstliche Intelligenz Initiative: „codecentric.AI“

Einführung in Computer Vision mit OpenCV und Python

<div><p>In unserem neuesten <a href="https://www.youtube.com/codecentricAI" target="_blank">codecentric.AI-Video</a>  geben wir eine kurze <a href="https://youtu.be/Quhy_yAz5OM" target="_blank">Einführung in Amazon SageMaker</a>  und zeigen, wie man damit schnell und einfach ein Bildklassifikationsmodell trainieren kann, das Brillenträger von Nicht-Brillenträgern unterscheidet.</p><div><div><div> <img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/3vetaHzSeOHE1fJOevJw" alt="YouTube" /><div><p>Mit dem Laden des Videos akzeptieren Sie die Datenschutzerklärung von YouTube.<br /><a href="https://policies.google.com/privacy" target="_blank">Mehr erfahren</a> <br /></p><p><a href="#">Video laden</a> </p><p> <small>YouTube immer entsperren</small></p></div></div></div><div></div></div><div><h2>Was ist Amazon SageMaker</h2><p><a href="https://aws.amazon.com/de/sagemaker/">Amazon SageMaker</a>  ist Teil der Amazon Web Services (<a href="https://www.codecentric.de/partner/aws">AWS</a> ) und</p><blockquote><p>“ist eine vollständig verwaltete Plattform, die es Entwicklern und Datenwissenschaftlern ermöglicht, schnell und einfach Machine-Learning-Modelle jeder Größenordnung zu erstellen, zu trainieren und zu implementieren.” <a href="https://aws.amazon.com/de/sagemaker/" target="_blank">https://aws.amazon.com/de/sagemaker/</a> </p></blockquote><p>SageMaker ist ein vollständig verwalteter Service, der den gesamten Machine-Learning-Prozess – vom Erstellen, über das Trainieren, bis hin zum Deployment von Machine-Learning-Modellen – so einfach wie möglich machen soll. Verfügbar ist SageMaker in Irland und in den USA.</p><p><img loading="lazy" width="720" height="405" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/OzpOKq3sQu4ibr2JLQwc" alt="SageMaker Machine Learning Prozess" /></p><p>Dabei werden viele der “alten Bekannten” aus der <a href="https://www.codecentric.de/partner/aws">AWS</a> -Umgebung in den Ablauf integriert verwendet: Trainingsdaten, Modellartefakte und Modellparameter werden in <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/gs-config-permissions.html" target="_blank">S3-Buckets</a>  gespeichert, wir können <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/gs-account.html#gs-account-user" target="_blank">IAM-User mit SageMaker-Berechtigung</a>  erstellen, und die <a href="https://docs.aws.amazon.com/cli/latest/reference/sagemaker/create-notebook-instance.html" target="_blank">Modelle laufen auf EC2-Instanzen</a> . Die Vorbereitung der Daten sowie das Trainieren selbst erfolgen in <a href="https://jupyter-notebook.readthedocs.io/en/stable/" target="_blank">Juypter Notebooks</a> , die in einer Notebook-Instanz laufen. Hierfür können grundsätzlich zwei Methoden verwendet werden: <em>boto3</em>, die <a href="https://aws.amazon.com/de/sdk-for-python/" target="_blank">AWS SDK für Python</a>  oder eine <a href="https://github.com/aws/sagemaker-python-sdk" target="_blank">high-level Pythonbibliothek</a> . Jupyter Notebooks hießen früher iPython Notebooks und wurden als Open-Source-Alternative zu Wolfram Notebooks entwickelt; die Notebooks laufen im Web Browser und kombinieren Code (in der Regel Python aber auch Kernel für R, Julia, oder andere Sprachen können eingerichtet werden) mit Markdown-Elementen und zeigen den Output direkt unterhalb der Code-Chunks an. Für die gestartete SageMaker-Instanz öffnet sich ein Dashboard, das alle Dateien in unserer Instanz anzeigt und von dem aus wir auch neue Notebooks erstellen können. Es gibt eine Reihe von “fertigen” Kernels, die viele der gängigen Datenverarbeitungs- und Analysepakete schon vorinstalliert haben (zum Beispiel conda für Python3 oder Python2, für TensorFlow oder MXNet). Deployen können wir ein fertiges Modell dann mit den <a href="https://docs.aws.amazon.com/de_de/sagemaker/latest/dg/how-it-works-hosting.html" target="_blank">Amazon SageMaker Hosting Services</a> .</p><p>Für das Trainieren von Modellen kann aus verschiedenen Algorithmen gewählt werden. Eine Übersicht gibt es <a href="https://docs.aws.amazon.com/de_de/sagemaker/latest/dg/algos.html" target="_blank">hier</a> . SageMaker hat gängige Machine-Learning-Algorithmen reimplementiert um Parallelisierung zu ermöglichen bzw. zu vereinfachen. So kann effektiv und schnell auch mit größeren Datenmengen gearbeitet werden. Wer speziellere Anwendungsfälle hat, kann auch eigene Algorithmen nutzen; die können dann als Docker Image zur Verfügung gestellt werden. Auch ermöglicht SageMaker sehr gute Skalierbarkeit, da die Trainings- und Hostinginstanzen jederzeit angepasst werden können, um mit größeren Datenmengen umgehen zu können. Auch können mehrere Modelle gleichzeitig deployed werden und somit produktiv laufen.</p></div><div><h2>Was brauche ich, um dem Video-Tutorial folgen zu können?</h2><p>Da SageMaker in die <a href="https://www.codecentric.de/partner/aws">AWS</a> -Welt eingebettet ist, benötigt man zunächst einen AWS Account. Das Registrieren bei <a href="https://www.codecentric.de/partner/aws">AWS</a>  ist an sich kostenlos, man braucht nur eine E-mail-Adresse, Passwort und Kreditkarteninformationen, um genutzte Services abrechnen zu können. Die ersten zwei Monate kann SageMaker kostenlos getestet werden. Aber Achtung: Das Aktivieren des Kontos kann bis zu 24 Stunden dauern!</p><p>Empfohlen wird, alle Analysen und Co. nicht mit dem Root User zu machen, sondern für jedes Projekt dedizierte IAM-User anzulegen und diesen die entsprechenden Rechte zu geben. Der Grund ist schlicht, dass dieses Vorgehen größere Sicherheit und Kontrolle gibt. Für SageMaker benötigt der IAM-User die sogenannten „AmazonSageMakerFullAcess“-Rechte. Der <a href="https://github.com/codecentric/codecentric.AI/blob/master/simple_deeplearning_aws_sagemaker/glasses-classifier.ipynb" target="_blank">komplette Code</a>  ist hier in Python geschrieben, Grundkenntnisse in Python sind also empfehlenswert. Außerdem ist es hilfreich, wenn man schon mal mit Jupyter gearbeitet hat (aber selbst wenn nicht, erschließt sich Jupyter sehr schnell).</p></div><div><h2>Kosten</h2><p>Die einzelnen Preise sind stark von der Menge der Daten, der genutzten Instanzen und der Trainingszeit abhängig. Auch für das Deployment und Abfragen der deployten Modelle fallen Kosten an. Eine Übersicht ist <a href="https://aws.amazon.com/de/sagemaker/pricing/" target="_blank">hier</a>  zu finden.</p></div><div><h2>Brillen-Klassifikation mit SageMaker</h2><p>In dem Video zeigen wir, wie wir einen Image-Classifier für einen Datensatz mit <a href="https://www.codecentric.de/ueber-uns/team/">codecentric-Mitarbeiterfotos</a>  entwickelt haben. Unser Datensatz enthält 460 Bilder. Wir verwenden ein tiefes neuronales Netz. Mit Transfer Learning wird es so traininert, dass es unsere Bilder in die Klassen „Brille“ und „keine-Brille“ einteilt. Transfer Learning macht es möglich, dass wir selbst mit diesem kleinen Datensatz gute Ergebnisse erzielen können. Für SageMaker gibt es eine Reihe von Beispiel-Notebooks, die zeigen, wie SageMaker verwendet werden kann. Diese Demo orientiert sich an diesen Beispiel-Notebooks (speziell an: <strong>Image-classification-lst-format.ipynb</strong>). Unser fertiges SageMaker Notebook ist <a href="https://github.com/codecentric/codecentric.AI/blob/master/simple_deeplearning_aws_sagemaker/glasses-classifier.ipynb" target="_blank">hier bei Github zu finden</a> .</p><p><img loading="lazy" width="1582" height="902" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/CUcLOqLrTxUKBMyoI4gg" alt="codecentric-Mitarbeiterfotos" /></p><p>In dem Video-Tutorial demonstrieren wir, wie wir zunächst die Bilder vorbereiten und für das Modell zugänglich machen. Dann teilen wir sie in Trainings-, Validierungs- und Testdaten auf. Im nächsten Schritt definieren und konfigurieren wir den TrainingsJob. Der Anfrage an die CreateTrainingJob API müssen wir dafür folgende Informationen mitgeben:</p><ul><li>den Algorithmus (als Pfad zum entsprechenden Container)</li><li>die Hyperparameter</li><li>Input- & Output-Konfiguration (als Pfad zu den entsprechenden S3-Buckets und Ressourcen)</li></ul><p>Die Ergebnisse des Trainings, also das Modell plus Konfiguration, sind in unserem S3-Bucket gespeichert.</p><p>Und schließlich konfigurieren wir den Endpunkt für das fertige Modell und deployen ihn. Denn um Vorhersagen mit einem Modell machen zu können, müssen wir es zuerst deployen; das passiert in drei Schritten:</p><ul><li>Modelle erstellen (Hier geben wir den Pfad zum S3-Bucket an, in dem die Trainingsergebnisse gespeichert wurden.)</li><li>Endpunkt konfigurieren</li><li>Endpunkt erstellen (Wenn der Endpunkt erstellt ist, werden die entsprechenden Machine-Learning-Instanzen hochgefahren, das Modell wird deployed und der Endpoint kann nun angesprochen werden.)</li></ul><p>So können wir per API auf unser Modell zugreifen und die Testbilder evaluieren.</p><p>Wenn wir unsere TrainingsJobs, Instanzen etc. überwachen wollen (was in der Praxis immer eine gute Idee ist), müssen die entsprechenden Logs in der Erstellung der Buckets und Instanzen aktiviert sein. Dann können wir die Ergebnisse der Log-Files im SageMaker Dashboard oder in Amazon CloudWatch anzeigen lassen.</p><p>Am Ende müssen wir noch “aufräumen”; das heißt, alle Ressourcen, die wir nicht mehr benötigen (wie S3-Buckets, Notebook-Instanzen, Modelle und Endpunkte), sollten wir stoppen oder löschen. Ansonsten fallen dafür weitere Kosten an.</p></div><div><h2>Weiterführende Informationen und Source Code:</h2><p>Amazon SageMaker Dokumentation: <a href="https://aws.amazon.com/de/documentation/sagemaker/" target="_blank">https://aws.amazon.com/de/documentation/sagemaker/</a> <br /> Link zum Source Code: <a href="https://github.com/codecentric/codecentric.AI/blob/master/simple_deeplearning_aws_sagemaker/glasses-classifier.ipynb" target="_blank">https://github.com/codecentric/codecentric.AI/blob/master/simple_deeplearning_aws_sagemaker/glasses-classifier.ipynb</a> </p></div></div>

Simple Deep Learning mit Amazon SageMaker

<p>Jeder kennt sie, keiner mag sie: Die Rede ist von Cookie-Bannern, die gerade im Laufe des Jahres 2020 immer lästiger geworden sind. Eine echte Plage. Grundsätzlich steckt dahinter eine gute Idee, nämlich die Benutzer*innen davor zu schützen, dass sie getrackt werden. Leider ist die Umsetzung in der Realität aber alles andere als benutzerfreundlich. Inzwischen liest doch kaum noch jemand ernsthaft die Cookie-Banner durch, sondern versucht diese möglichst schnell wegzuklicken. Viele sind bereits auch so genervt, dass sie Seiten mit aufdringlichen Bannern einfach direkt wieder verlassen.</p><p><img width="872" height="742" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qsxrNMmdRBqeRPEd7YrJ" alt="Sorry, NoMoreCookies" /></p><p>Wir haben es uns zur Aufgabe gemacht, etwas dagegen zu unternehmen. Denn so kann es nicht weiter gehen. Wir müssen respektvoll mit den Daten unserer Benutzer*innen umgehen. Wir müssen gemeinsam überlegen und Wege finden, wie wir Webseiten von morgen betreiben können und wollen – DSGVO-konform, datensparsam und ohne heimliches Tracking. Einige heute verbreitete Web Analytics Tools tun aber genau das Gegenteil. Sie sammeln so viele Daten wie möglich und bringen diese über Domaingrenzen hinweg in Bezug.</p><p>Daher haben wir damit begonnen ein eigenes Tool zu entwickeln, das völlig ohne Cookies auskommt und keine persönlichen Daten erhebt. Die Zugriffe werden dabei durch Hash-Funktionen anonymisiert und so aggregiert, dass man daraus keine einzelnen Sessions mehr rekonstruieren kann. Die Daten sind also völlig anonym. Wir speichern keine IP-Adressen oder ähnliches. Wir beschränken uns dabei auf wenige essentielle Metriken, die man ohne Zustimmung erfassen darf. Kommt ein/e Benutzer*in am nächsten Tag wieder, bekommt er/sie einen völlig anderen Hash und ist mit vorherigen Zugriffen nicht mehr in Verbindung zu bringen. <b>Simple Web Analytics made in Germany</b>. Wer uns auf unserer Mission gegen die Cookie-Banner begleiten möchte, findet unseren Service unter https://sw.report/.</p><h3>Tausche Daten gegen Geld?!</h3><p>Einige große Analyse-Tools kann man vermeintlich kostenlos verwenden. Doch der Anbieter muss natürlich etwas damit verdienen – es geschieht nicht aus Nächstenliebe. Ein gängiges Geschäftsmodell ist, die gesammelten Daten auszuwerten und zu verkaufen. Im Bereich Online Marketing lässt sich damit sehr viel Geld verdienen. Doch die Zeiten werden schwieriger werden. Das heimliche Ausspähen von Benutzern ist kein Kavaliersdelikt mehr, und so werden inzwischen empfindliche Strafen verhängt, wenn Firmen das Datensammeln übertreiben.</p><p>Wir haben für unseren Service ein anderes Geschäftsmodell gewählt. Wir werten die Daten nicht für andere Zwecke aus, und wir verkaufen keine Daten weiter. Dafür verlangen wir einen monatlichen Betrag, der sich in der Höhe im wesentlichen daran orientiert, ob man eine Webseite mit sehr vielen Zugriffen hat oder nur einen kleinen Webauftritt mit ein paar 1000 Klicks im Monat betreibt. Zum Ende des Jahres 2020 bieten wir auch die Möglichkeit, unser Tool 30 Tage kostenlos zu testen. Klingt nach einem fairen Angebot? Dann schau bei uns vorbei und empfehle uns weiter. Wir wünschen eine Menge leckere Weihnachts-Cookies und freuen uns auf den Besuch: https://sw.report/</p>

Simple Web Reports, #NoMoreCookies und #BanTheCookieBanners

<p><img width="700" height="241" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/xdHv9XWhQiWa18pFIblq" alt="bootcamp header" /></p><p>Im letzten Jahr haben wir bei codecentric eine AI-Initiative gestartet. Wir haben uns zum Ziel gesetzt, einen Online-Kurs zum Thema <strong>Machine Learning und künstliche Intelligenz</strong> in deutscher Sprache zu entwickeln. Natürlich gibt es bereits mehrere sehr gute Online-Kurse zu diesen Themen im Internet – allerdings sind die meisten auf Englisch und haben einen eher akademischen Schwerpunkt bzw. Verlauf. Aus eigener Erfahrung haben wir gemerkt, dass es frustrierend sein kann, wenn man sich zunächst (über Monate) mit den theoretischen Grundlagen beschäftigt und dabei kein Gefühl dafür bekommt, wie man diese Technologien praktisch einsetzen könnte. Wir sind der Meinung, dass die verfügbaren Tools und Frameworks heute so zugänglich geworden sind, dass man sie auch ohne großen Data-Science-Hintergrund nützlich in eigenen Projekten anwenden kann. Unser AI Bootcamp soll daher einen komprimierten und praktischen Einstieg geben und vor allem Entwickler ansprechen, die sich den Themen zunächst einmal durch Code-Beispiele nähern wollen.</p><p>Wenn du dir das AI Bootcamp anschauen möchtest ohne weiter zu lesen, findest du hier den Link: bootcamp.codecentric.ai.</p><p>Bei codecentric gibt es das Konzept der „20%“- bzw. „+1“-Zeit. Das bedeutet, dass Mitarbeiter/*/innen etwa einen Tag pro Woche Zeit haben, um sich eigenen Projekten oder der persönlichen Weiterbildung zu widmen. Viele Inhalte dieses Kurses sind in dieser Zeit entstanden – wir versuchen nützliche (und kostenlose) Inhalte zu erstellen und damit einen Beitrag für eine wachsende deutschsprachige Machine-Learning-Community zu leisten. Wir freuen uns über jegliches Feedback. Was können wir verbessern? Was hat dir gut gefallen? Welche Themen würdest du gerne im Kurs sehen?</p><h2>Was erwartet dich in dem Kurs?</h2><p>Wir haben versucht, die Kurskapitel auf das Wesentliche zu komprimieren und möglichst kurz zu halten, so dass man Schritt für Schritt immer wieder eine kleine Einheit nebenbei absolvieren kann. Am Anfang haben wir die Themen mit Absicht etwas vereinfacht, um einen „sanften Einstieg“ zu ermöglichen. Wir beginnen mit einer kurzen Einführung und wie man eine einfache Entwicklungsumgebung einrichtet. Dann kommen wir über Basics zum Thema Maschine Learning, Basics im Bereich Computer Vision zum Schwerpunkt-Thema: <strong>„Neuronale Netze und Deep Learning“.</strong></p><p>In den einzelnen Kapiteln haben wir jeweils ein „Erklär-Video“ erstellt, in dem wir Begriffe und Theorie behandeln. Darauf folgt dann ein Jupyter Notebook mit Code-Beispielen und Übungsaufgaben. Um an dem Kurs teilnehmen zu können, solltest du etwas Erfahrung als Software-Entwickler haben und Basiskenntnisse in Python mitbringen. Weitere Vorkenntnisse sind eigentlich nicht notwendig.</p><p>Der Kurs ist „Work in Progress“ – wir planen noch weitere Kapitel und Themen. Vor allem wollen wir dies anhand von Praxis-relevanten Beispielen tun.</p><h2>Ein paar Einblicke</h2><h3>Machine Learning Basics</h3><p><img loading="lazy" width="700" height="486" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/LcPBbSPPS4CH5rSAoJnL" alt="machine learning basics" /></p><p>Im Kapitel „Machine Learning Basics“ erklärt das AI Bootcamp grundlegende Begriffe wie Kosten-/Loss-Funktion, Klassifikation, Regression, Recall, Cross-Validation und vieles mehr. Wenn du noch keine Erfahrung im Bereich Machine Learning hast, solltest du dieses Kapitel nicht überspringen.</p><h3>Computer Vision Basics</h3><p><img loading="lazy" width="700" height="497" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qSLSMsq1Q3i2973vrhcX" alt="computer vision basics" /></p><p>Im Kapitel „Computer Vision Basics“ erklären wir, wie ein Computer ein Bild „sieht“ und einfache Methoden um Bilder zu manipulieren. Am Beispiel einer „Computer-Vision-Drohne“ zeigen wir einen einfachen Personen-Detektor und wie man Algorithmen miteinander kombinieren kann, um eine Person in einem Bild zu fokussieren.</p><h3>Neuronale Netze und Deep Learning</h3><p><img loading="lazy" width="700" height="502" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/2TufSB14SbWuJV0tBtPN" alt="Katzen Detektor" /></p><p>Im Kapitel „Neuronale Netze und Deep Learning“ wollen wir gemeinsam eine kleine Web-Applikation bauen, die einer Katze eine Sonnenbrille aufsetzen kann. Dazu trainieren wir ein eigenes Neuronales Netz und verwenden das entstehende Modell, um die Augen der Katze in einem Bild zu finden. Wir kombinieren dies mit Computer Vision Tools, um der Katze eine Sonnenbrille an die richtige Stelle zu setzen.</p><p><img loading="lazy" width="700" height="569" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/kc5lv3FRkCoq4pUolsLw" alt="einfaches neuronales Netz" /></p><p>Vorher schauen wir uns genauer an, was neuronale Netze sind und welche Rechenoperationen auf unterster Ebene ausgeführt werden, damit ein solches Netz lernen kann. Wir geben einen Überblick darüber, welche Frameworks und Tools es gibt und schreiben einen eigenen Trainings-Loop.</p><p><img loading="lazy" width="700" height="464" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/jXgBiFV3S3ysGsFlZoLL" alt="neuronales Netz" /></p><p>Anschließend geben wir einen Überblick welche Arten von neuronalen Netzen es gibt und für welche Anwendungsfälle welche Architektur geeignet ist.</p><p><img loading="lazy" width="700" height="479" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/CRXnOgFPQU604kntZ5JY" alt="CNN Überblick" /></p><p>Schließlich zeigen wir im AI Bootcamp noch, was Transfer Learning ist und welche Vorteile sich dadurch ergeben, und implementieren dann unseren Katzen-Augen-Detektor mit der Sonnenbrillen-App.</p><p><img loading="lazy" width="700" height="462" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/sQ1j4NUSSIW84m5csaNO" alt="cat dataset" /></p><h3>NLP – Natural Language Processing</h3><p>Im Kapitel „NLP“ oder „Natural Language Processing“ geben wir eine Einführung in das Thema Sprach-Verarbeitung und zeigen praktische Beispiele wie du diese einsetzen kannst, um automatisch gesprochene oder geschriebene Sprache „verstehen“ zu können.</p><p><img loading="lazy" width="700" height="472" src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/az3s4m7GRuKRvZdH89Rz" alt="natural language processing" /></p><h2>Kann es los gehen?</h2><p>Wenn du jetzt unseren Kurs ausprobieren möchtest, dann kannst du dich hier:<br /> bootcamp.codecentric.ai kostenlos anmelden. Dort findest du auch ein Formular, um uns zu kontaktieren und einen Link zu unserer Slack Community, wo man sich zu den Kursinhalten austauschen und Fragen stellen kann. Wenn dir unser Kurs gefällt freuen wir uns, wenn du den Link mit Bekannten und Kollegen teilst. Natürlich sind wir auch dankbar für jedes Feedback. Zusätzlich zum Online-Kurs haben wir einen <a href="https://youtube.com/codecentricAI" target="_blank">Youtube-Channel</a>  eingerichtet, wo wir weitere Videos zum Thema künstliche Intelligenz veröffentlichen.</p><h2>Angebote für Firmen</h2><p>Zusätzlich zu unserem Online-Kurs bieten wir auch verschiedene Vor-Ort-Veranstaltungen für Firmen an. Weitere Infos dazu findest du ebenfalls auf unserer AI-Bootcamp-Seite. Weitere Infos zum Thema künstliche Intelligenz bei codecentric gibt es auch unter <a href="http://codecentric.ai" target="_blank">codecentric.AI</a> .</p><h2>Weiterführende Links</h2><p><a href="https://blog.codecentric.de/?s=machine+learning&x=0&y=0">Artikel zum Thema Machine Learning</a> <br /> <a href="https://blog.codecentric.de/?s=computer+vision&x=0&y=0">Artikel zum Thema Computer Vision</a> <br /> <a href="https://blog.codecentric.de/?s=nlp&x=0&y=0">Artikel zum Thema Natural Language Processing</a> </p>

codecentric.AI Bootcamp ist online!

Service Lead GenAI

<p>Usually, debugging issues related to TLS in a Java application involves setting the <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html" target="_blank"> debug flag</a>  <code>-Djavax.net.debug=ALL</code>. Unfortunately, the logs will then be flooded with debug entries, which makes troubleshooting the TLS communication difficult. Moreover, sometimes the problem can be solved <strong>just</strong> by decrypting the TLS traffic.</p><p>One way to achieve this is to use <a href="http://jsslkeylog.sourceforge.net/" target="_blank">jSSLKeyLog</a> , which is a Java Agent Library to log TLS session keys to a file that can be loaded into the <a href="https://www.wireshark.org/" target="_blank">Wireshark</a>  network protocol analyzer. Coupled with the recently improved <a href="https://www.wireshark.org/docs/dfref/k/kafka.html" target="_blank">Kafka Protocol support</a>  in Wireshark, this makes decrypting TLS traffic to/from Kafka captured using the <a href="https://www.tcpdump.org/" target="_blank">Tcpdump</a>  packet sniffer easy.</p><h2><a> Procedure</a></h2><p>Instrument the Kafka Broker and/or Kafka Client with the jSSLKeyLog agent by adding the JVM flag</p><div>

```text
-javaagent:/path/to/jSSLKeyLog.jar==/path/to/ssl-key.log
```

</div><p>when starting the respective process. The Kafka Broker can be easily instrumented by appending the flag to the <code>KAFKA_OPTS</code> Environment Variable.</p><p>Then, use the Tcpdump packet sniffer to capture the traffic to/from the Kafka Broker SSL listener, for example by executing the command</p><div>

```bash
sudo tcpdump -i eth0 -N -A 'port 9093' -w /path/to/trace.pcap
```

</div><p>on the machine running the Kafka Broker (adjust the interface, port or path to the packet capture file accordingly).</p><p>After the capture finished, open the packet capture file <code>trace.pcap</code> using Wireshark. Without configuring the session key log, the TLS payload will be displayed as “Application Data”.</p><p><a href="https://blog.codecentric.de/files/2020/01/jsslkeylog-1.png"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/qYeAZh3jQyaDiHgB1Ldb" alt="open tcpdump with Kafka TLS traffic in Wireshark" /></a> </p><p>Next, configure the location of the session key log generated by jSSLKeyLog by choosing <strong>Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename</strong> in Wireshark.</p><p><a href="https://blog.codecentric.de/files/2020/01/jsslkeylog-2.png"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/AJ5IlrouRZivoJqmEZgm" alt="set session key log in Wireshark" /></a> </p><p>This allows us to inspect the decrypted TLS payload. To also take advantage of the Kafka Decoder in Wireshark, select <strong>Analyze > Decode As > Kafka</strong> as the dissection protocol.</p><p><a href="https://blog.codecentric.de/files/2020/01/jsslkeylog-3.png"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/A3xrs4W5QyKxIBjel294" alt="configure Kafka Decoder in Wireshark" /></a> </p><p>As a result, Wireshark will display the decoded Kafka API Request/Response pairs.</p><p><a href="https://blog.codecentric.de/files/2020/01/jsslkeylog-4.png"><img src="https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/KxpEb8oIS7Gy4tCfMNL5" alt="decoded Kafka API Requests and Responses" /></a> </p><h2>Demo</h2><p>The accompanying <a href="https://github.com/tudrescu/kafka-wireshark-debug-workshop" target="_blank">GitHub repository</a>  provides a quick way to test the procedure described above using Docker Compose.</p><h3>Requirements</h3><p>The demo uses versions of <a href="https://www.wireshark.org/" target="_blank">Wireshark</a>  and <a href="https://github.com/edenhill/kafkacat" target="_blank">kafkacat</a>  not (yet) available in the standard package repositories.</p><ul><li>the attached <code>Makefile</code> will build several Docker containers with the needed versions by downloading the source code and compiling the required binaries.</li><li>the <a href="http://jsslkeylog.sourceforge.net/" target="_blank">jSSLKeyLog</a>  Java Agent will also be built from source and the artifact saved in the <code>tools</code> directory. The directory will then be mounted inside the Docker container running the Kafka Broker and the agent attached via<div>

```text
KAFKA_OPTS: "-javaagent:/tmp/tools/jSSLKeyLog.jar==/tmp/ssl-key-log/output.log"
```

</div></li></ul><p>To build all the required components, simply run</p><div>

```bash
make build-all
```

</div><p>inside the project directory.</p><p><strong>Warning</strong>: building the required Docker containers requires some time as Wireshark in particular is a complex application.</p><h3>Running the demo</h3><ul><li>clone the repository<div>

```bash
git clone https://github.com/tudrescu/kafka-wireshark-debug-workshop.git
cd kafka-wireshark-debug-workshop
```

</div></li><li>bootstrap the environment<div>

```bash
make build-all
```

</div></li><li>create a local CA, Server and Client certificates<div>

```bash
make certs
```

</div></li><li>launch the Docker Compose stack<div>

```bash
docker-compose up -d
```

</div></li><li>start a Docker container running tcpdump and attach to the container running the Kafka Broker, see also <a href="https://rmoff.net/2019/11/29/using-tcpdump-with-docker/" target="_blank">using tcpdump with Docker</a> .<div>

```bash
docker run \
  --tty \
  --net=container:kafka-1 \
  -v ${PWD}/work/tcpdump-trace:/tmp/tcpdump-trace \
  docker-tcpdump:latest \
  tcpdump -N -A 'port 9093' -w /tmp/tcpdump-trace/trace.pcap
```

</div></li><li>enter the kafkacat Docker container and produce some messages<div>

```bash
docker-compose exec kafkacat bash
kafkacat -b kafka-1:9093 -L -F /tmp/certs/kafkacat.conf
kafkacat -b kafka-1:9093 -t new_topic -P -F /tmp/certs/kafkacat.conf
```

</div></li><li>stop the packet capture using tcpdump</li><li>open the packet capture using Wireshark. For convenience, the Docker stack provides a containerized Wireshark which is made available in the browser using <a href="https://xpra.org/" target="_blank">Xpra</a> . Access the Wireshark via the browser at<div>

```text
https://localhost:14500/?username=wireshark&password=wireshark
```

</div></li><li>configure Wireshark to decode the Kafka TLS traffic as detailed <a href="#procedure">above</a> . The session key log and the packet capture are saved in the <code>${PWD}\work\ssl-key-log</code> and <code>${PWD}\work\tcpdump-trace</code> directories, respectively. The directories are mounted in read-only mode inside the Wireshark container under the <code>/home/wireshark/work</code> path.</li><li>stop and destroy the stack when finished<div>

```bash
docker-compose down -v
```

</div></li></ul><h2>Summary</h2><p>With this approach, we can easily capture and decode TLS traffic to troubleshoot Kafka related issues by inspecting the Kafka protocol requests and responses. The improved support for the dissection of the Kafka protocol available in the latest Wireshark releases might also prove valuable when trying to understand how Kafka works internally.</p><h2>References</h2><ul><li><a href="https://kafka.apache.org/protocol" target="_blank">Kafka protocol guide</a> </li><li>Kafka 2.3+ dissesction support in Wireshark</li><li><a href="https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=15988" target="_blank">Bugzilla ticket with sample captures for Kafka 2.3+</a> </li><li><a href="https://nostarch.com/packetanalysis3" target="_blank">Practical Packet Analysis, 3rd Edition</a> </li><li><a href="https://rmoff.net/2019/11/29/using-tcpdump-with-docker/" target="_blank">Using tcpdump with Docker</a> </li><li><a href="https://github.com/ffeldhaus/docker-wireshark" target="_blank">Wireshark Web Container Image</a> </li></ul>

Decrypt TLS traffic to Kafka using Wireshark

IT Consultant and Software Engineer

Software Architecture Consultant

IT-Security Consultant

Content & Social Media Managerin

Karlsruhe | Team

Vielen Dank für deine Anfrage. Ich melde mich bei dir.

Vielen Dank für deine Anmeldung. Eine E-Mail zur Bestätigung ist zu dir unterwegs.

Diese Seite ist eine Vorschau. \nKlicken Sie hier\n, um den Vorschaumodus zu verlassen.

Zu deiner Suche haben wir leider keine passenden Blogartikel gefunden. 

Du hast noch Fragen zu diesem Thema? Dann sprich mich einfach an.

Du hast Fragen oder möchtest dich mit mir austauschen? Gern - trag einfach deine Kontaktdaten ein.

Die \nDatenschutzerklärung\n der codecentric AG habe ich zur Kenntnis genommen und bestätige das mit der Absendung des Formulars.*

Entdecke spannende weiterführende Themen und lass dich von der codecentric Welt inspirieren.

Ich möchte eure Inhalte abonnieren. Diese Einwilligung kann ich jederzeit widerrufen bzw. den Blog Newsletter abbestellen. Die Datenschutzerklärung habe ich gelesen. Ich bin damit einverstanden, dass die codecentric AG, Hochstr. 11, 42697 Solingen, mich über eigene Dienstleistungen, branchenrelevante Events, Webinare und Publikationen der codecentric AG über die von mir angegebene E-Mail-Adresse informiert. Meine Einwilligung ist freiwillig. Ich kann meine Einwilligung jederzeit mit Wirkung für die Zukunft widerrufen (z.B. über den hierfür vorgesehenen Abmeldelink in unserem Newsletter, per E-Mail an datenschutz@codecentric.de oder per Post an die oben genannte Adresse der codecentric AG) erklären. Weitere Hinweise zur Verarbeitung meiner personenbezogenen Daten und zu meinen Rechten als betroffene Person finde ich hier: \nhttps://www.codecentric.de/datenschutzerklaerung\n

Bitte wähle deine/n gewünschte/n Standort/e aus.

Willkommen im Team Karlsruhe

Jetzt für unseren Newsletter anmelden