Thanks for doing some research into this.

I think the approach I would take is to use Strophe.jingle (or something similar) with WebRTC.

Would be a nice feature to have. Ideally a company that needs this feature would fund the development for it.

I found this libjingle adaption for javascript https://github.com/otalk/jingle.js
libjingle is a library that makes it possible to use audio and videochat over xmpp and clients like pidging are already using libjingle. So maybe this could help you to implement that feature.

Any news?

There are no plans to add audio/video support, but pull requests are welcome if someone wants to take up the challenge.

maybe now webRTC would be a good technology to use. But yes it needs some effort to implement.

If there is a person who can do this and embed the code for audio / video as in pix-art messages, he is ready to sponsor this project.

@free-14 you mean Conversations, since Pix-Art is just a fork, right? Also... that's Java on Android... This is Javascript.

I have a working conversejs plugin for jingle audio/video calls that works with the jingle implementation in Conversations. However, it has outstanding issues and requires a make-over to remove the out-dated jquery xml parsing. I am not free to work on it at the moment, but for anyone who has the time, space and ability, here is the link

https://github.com/igniterealtime/pade/blob/version-1.xx/docs/inverse/plugins/jinglecalls.js

@free-14 you mean Conversations, since Pix-Art is just a fork, right? Also... that's Java on Android... This is Javascript.
Yes, you noticed correctly. It's just that I'm already used to the design and additional features of pix-art. )

There is another ready-made solution that works well in nextcloud - Jsxc. It is open source. I think this decision is not bad when combined.

Implementation of this feature is now part of our 2022 Google Summer of Code application. See here: https://wiki.xmpp.org/web/Google_Summer_of_Code_2022#Support_Audio/Video_calls_in_Converse

I will be acting as mentor for this project.

I've done a bit of research and want to write down my thoughts and findings as a resource for myself and whoever will be doing the implementation.

Note: For now I'm just going to do a brain dump, and I'll likely update this comment multiple times over an extended period.

Introduction

Audio/Video calls are done outside of XMPP via WebRTC. XMPP (in the form of the Jingle protocol extension) is merely used as a means to initiate and terminate an audio/video session. In plain language, this means that Jingle is used to call the other person (AFAIK that's why it's called "Jingle", like ringtone), and that person can then pick up, or reject the call. If they pick up the call, either person has the option to end the call at any time.

Making the call and accepting, rejecting or ending a call are all session management operations that are done by sending XMPP stanzas between two clients.

UI components

So, from a UX perspective, we need the following UI elements:

• A call button in the 1:1 chat toolbar, which can be pressed to call someone.
• A modal to show incoming calls with buttons to accept or reject the call.
• A UI component which shows finished (historical) calls in the chat history
• A popup window which shows an ongoing call with buttons to end the call, mute yourself and turn video on or off.

Specifications

The main XEPs related to this feature are:

• XEP-0166 Jingle - Describes how a Jingle session is initiated, accepted, rejected or terminated via IQ stanzas.
• XEP-0167: Jingle RTP Sessions
• XEP-0353: Jingle Message Initiation - Describes how a message stanza can be used to initiate a Jingle session, which addresses many of the shortfalls and problems with XEP-0166.

How to start:

We need to create an new plugin called jingle (eventually it could be split up into two pugins, one without UI elements for @converse/headless and another for the UI).

Then I would start by writing tests, and I would start with [XEP-0353: Jingle Message Initiation](XEP-0353: Jingle Message Initiation). I would write a test where the user receives an incoming Jingle (XEP-0353) message and then checks that the right things happen (e.g. that the user is informed of the incoming call, and if they accept the call, that the right stanzas are sent out).

Multiple tests could be written here, to go through the various scenarios explained in the XEPs.

I would first write all the tests for the Jingle stanza traffic (and the associated UI elements) before worrying about WebRTC and actually setting up the audio/video calls.

Once the tests (and resultant implementations) are done, we can look at actually setting up the WebRTC session in a popup.

Other implementations that can be used as reference

Check the license before just blindly copying code. MIT, BSD and MPL are ok, but GPL isn't. When reusing big chunks of code, make sure to include copyright and licencing information.

• jinglecalls.js
• xabber-web
• jsxc
• strophe.jingle
• jingle.js

JSXC has an example implementation that also works for group calls and with an SFU. They wrote down an overview here: https://www.jsxc.org/blog/2021/08/31/A-group-call-proposal.html

@poVoq thanks, group calls are outside of the scope of this issue and the GSoC project. We can make a separate feature request for it.

@PawBud

As discussed in our last video call, here is the information you need regarding the next step of showing information about calls in the chat history.

The chat history is rendered via the component converse-message-history that's defined here:
https://github.com/conversejs/converse.js/blob/0cfe2a18afb334066abc2935020cc3d026334bd9/src/shared/chat/message-history.js

On line 34, you'll see it triggers a hook via (api.hook) and the name of the hook is in a variable called template_hook which comes from the message model (this.model.get('template_hook')) which will be rendered by the renderMessage function.

See here:

What this code does, is it allows plugins to control which template is rendered for a particular message. To do so, you have to make sure that there is an attribute template_hook stored on the message object. The value of that attribute is then the name of the hook that gets triggered. You can then register a handler for that hook, which returns a template that renders the message.

So what you can do, is when you parse an incoming message to see if it's a Jingle message (in parseJingleMessage), you can add another attribute to the attrs object, namely template_hook: getJingleTemplate.

That attribute will then be stored on the message object that gets created after parsing has been completed, and so when renderMessage for that message is called in the MessageHistory component, the hook getJingleTemplate will be triggered.

The above will handle incoming messages.

For outgoing messages, you set the template_hook attribute when you create the message object, here:

You can then create a handler for that hook which returns a template to render the jingle message in the chat history. This template you need to create yourself. You can look at the general message template for inspiration:
https://github.com/conversejs/converse.js/blob/0cfe2a18afb334066abc2935020cc3d026334bd9/src/shared/chat/templates/message.js

Your Jingle message template will have some similarities with regards to the markup it renders.

Once all of this is done, you should be able to see your Jingle messages rendered inside the chat history.