If you want your elements to be DOM elements, then I'd be curious to hear people's answers. On the other hand if you are good with the elements being data inside a canvas element (which you can convert to SVG if needed), then fabric.js will get you going really really fast because it's rich and already handles a lot of the editing needs (e.g. dragging, resizing, etc.). - Source: Reddit / 25 days ago
jsPlumb - jsPlumb is an advanced, standards-compliant and easy to use JS library for building connectivity based applications, such as flowcharts, process flow diagrams, sequence diagrams, organisation charts, etc.
p5.js - JS library for creating graphic and interactive experiences