HT-ZF9 API and discovery process

TheFC Company

@grolschie thanks but i have already try this. Here is my request, maybe another parameter is wrong? :
String customQuery = "M-SEARCH * HTTP/1.1" + "\r\n" + "HOST: 239.255.255.250:1900" + "\r\n" +
"MAN: "ssdp:discover"" + "\r\n" + "MX: 1\r\n"+ "ST: ssdp:all\r\n" + "\r\n";

@david tkanks it works !

grolschie

Hi. You need to escape the quote marks on the MAN line. i.e.

String customQuery = "M-SEARCH * HTTP/1.1\r\n" 
    + "HOST: 239.255.255.250:1900\r\n" 
    + "MAN: \"ssdp:discover\"\r\n"
    + "MX: 1\r\n"
    + "ST: ssdp:all\r\n" 
    + "\r\n";

That would most likely explain why the M-SEARCH received no responses. You might like to try using the ST line recommended by Sony now, instead of "ssdp:all", to see if that works.

TheFC Company

@grolschie it's already the case, it was removed by the copy and paste, sorry. With this query i can see my other upnp equipment...so i guess my model does not support it.

grolschie

If you have an Android device, I recommend an app called UPnP Tool. It finds UPnP devices, even my Sony AVR's API service (when I cannot!). You might find something else to query, like I did.

What character encoding are you encoding your String as, UTF-8?

grolschie

I'm wondering if anyone can point us to some example code for the implementation of M-SEARCH discovery process please? It would be handy to know if there's some extra massaging required for the Sony devices to respond. I hit a road block here and never solved it.

TheFC Company

@grolschie
Your tool is working ! Here is a screenshot :
HT-ZF9

I don't know for UTF-8, I'm using this upnp api: https://github.com/custanator/android-upnp-discovery

grolschie

It's not my tool. I wish it was, because then I'd know how to get my Sony AVR to respond to me properly. These Sony's are very picky at responding to M-SEARCH requests I think.

I looked at the the github project, and it's doing something similar to what I am doing and with the same results as me most likely. Maybe it is some Java weirdness? Here is the code in question.

Maybe some Sony person can comment? Perhaps @david please?

david

Unfortunately I have the same problem as you, the devices are very picky then it comes to respond to discovery.
Have had to use Wireshark to compare my messages to another application that worked to get it to work, and compare messages on bit level.
I usually do the same way as @grolschie and use "ssdp:all" since the devices seems more likely to reply to that.
Have also found that the devices in some cases can be very slow to respond so I had to use a longer timeout than expected and some times it take multiple tries.

grolschie

@david Thanks for the reply. Is there any way engineers can update existing products to fix this? Or to get an explanation from them of what magic is actually required to get a response from the device (the official way)?

david

@grolschie I sort of got the normal engineering answer "it works for me".

By the way here are my python script for discovery (haven't runned it for a while and think this version only works with python2). I'm using "ssdp:all" here since I was using it to test DLNA functions. I'm looking for both 'AVTransport', 'ContentDirectory' and 'ScalarWebAPI' functionality in the responce.

#!/usr/bin/env python
# encoding: UTF-8

import socket
import urllib2
import urlparse
import xml.etree.ElementTree as ET
import StringIO

SSDP_BROADCAST_ADDR = '239.255.255.250'
SSDP_BROADCAST_PORT = 1900
SSDP_BROADCAST_PARAMS = [
    'M-SEARCH * HTTP/1.1',
    "HOST: {0}:{1}".format(SSDP_BROADCAST_ADDR, SSDP_BROADCAST_PORT),
    'MAN: "ssdp:discover"', 'MX: 10', 'ST: ssdp:all', '', '']
SSDP_BROADCAST_MSG = '\r\n'.join(SSDP_BROADCAST_PARAMS)

def get_devices(timeout=10.0):
    """Find UPNP devices in current local network"""
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 4)
    s.bind(("", SSDP_BROADCAST_PORT + 10))

    s.sendto(SSDP_BROADCAST_MSG.encode('UTF-8'), (SSDP_BROADCAST_ADDR,
                                                  SSDP_BROADCAST_PORT))

    s.settimeout(timeout)

    devices = []
    while True:
        try:
            data, _addr = s.recvfrom(1024)
        except socket.timeout:
            break

        try:
            info = [a.split(":", 1)
                    for a in data.decode('UTF-8').split('\r\n')[1:]]
            device = dict([(a[0].strip().lower(), a[1].strip())
                           for a in info if len(a) >= 2])
            devices.append(device)
        except Exception:
            pass

    devices_urls = [{'location':dev['location'], 'type':dev['st']}
        for dev in devices if any(x in dev['st']
            for x in ['AVTransport', 'ContentDirectory', 'ScalarWebAPI'])]
    devices = [register_device(location_url) for location_url in devices_urls]

    return devices

def register_device(location_url):
    """Get device information"""
    device_xml_url = location_url['location']

    print "Get device xml from {0}".format(device_xml_url)

    device_xml = urllib2.urlopen(device_xml_url).read().decode('UTF-8')

    info = parse_xml(device_xml)

    location = urlparse.urlparse(device_xml_url)
    hostname = location.hostname

    friendly_name = info.find('./device/friendlyName').text

    if 'ScalarWebAPI' in location_url['type']:
        xpath = './device/X_ScalarWebAPI_DeviceInfo/X_ScalarWebAPI_BaseURL'
        action_url = info.find(xpath).text
    else:
        xpath = "./device/serviceList/service/[serviceType='{0}']/controlURL"
        path = info.find(xpath.format(location_url['type'])).text
        action_url = urlparse.urljoin(location_url['location'], path)

    uuid = info.find('./device/UDN').text

    device = {
        'location': location_url['location'],
        'hostname': hostname,
        'friendly_name': friendly_name,
        'action_url': action_url,
        'st': location_url['type'],
        'uuid':uuid
    }
    return device

def parse_xml(xml_string):
    """Removes namespaces and parse xml"""
    it = ET.iterparse(StringIO.StringIO(xml_string))
    for _, el in it:
        if '}' in el.tag:
            el.tag = el.tag.split('}', 1)[1]  # strip all namespaces
    return it.root

grolschie

Thank you, @david.

TheFC Company

I think I have found the github source for UPnP Tool:
https://github.com/bjtj/upnp-java
(It seems to be the same author)

I will get it a look later

grolschie

@TheFC-Company great find! I will have a more thorough look later. I'm no expert on datagrams, but I'm wondering if it has something to do with joining a multicast group vs merely sending a datagram to a multicast address - the latter is what I am doing and numerous other devices do respond.

Edit: the project also uses a DatagramChannel, which is something I'm unfamiliar with.

TheFC Company

@grolschie
I have sent an email to the UPnP Tool guy, he confirm for the github code, here is it's full reply:

M-SEARCH packet sent from UPnP Tool is like this and you can also check it or more packet logs by clicking "LOG…” Button on the main screen

If you need a java code sending M-SEARCH via datagram socket, please check this code
https://github.com/bjtj/upnp-java/blob/master/src/main/java/com/tjapp/upnp/SSDPMsearchSender.java
Above code is mine and very similar to UPnP Tool’s code

So the code that he use is:

String customQuery = "M-SEARCH * HTTP/1.1\r\n" 
    + "HOST: 239.255.255.250:1900\r\n" 
    + "MAN: \"ssdp:discover\"\r\n"
    + "MX: 3\r\n"
    + "ST: upnp:rootdevice\r\n" 
    + "USER-AGENT: Android/27 UPnP/1.1 UPnPTool/1.5.1"
    + "\r\n";

Unfortunately, that didn't help me to find the device. So we need to dig more in the datagram thing, but I'm not familiar too

grolschie

@TheFC-Company Thanks for the update. I also see that he uses an HttpHeader object rather than just construct a String. Then puts that into a byte array. I wonder why?

grolschie

@TheFC-Company any joy? I have had a further look. So the format of that project's SSDP packet is pretty much the same. The HttpHeader object is just a class in the same project that stores key-value pairs.

Functionally, the only major difference I can see is that he is using a DatagramChannel instead of a DatagramSocket. Hmmm.....

grolschie

I found and fixed a bug in my detection code. I managed to finally get a response from the receiver using the Android emulator, but not from my actual phone. So for now "ssdp:all" shall remain my ST field value.

grolschie

Hi @david. The weirdest thing. Using "ST: urn:schemas-sony.com:service:ScalarWebAPI:1" is now strangely working for me!!!??? I have no idea why! My original search string works (posted back on StackOverflow)!

I'm wondering if the most recent firmware update for the STR-DN1080 changed something. I'm reluctant to roll out the corrected code in case it actually did and some users haven't updated their firmware. The release notes didn't mention anything.

Failing that, a library update at my end or something...

david

@grolschie glad to hear that it is working for you now. Don't know if anything has changed in the last firmware and have limited access to devices right now, but tested on SRS-ZR5 and HT-ZF9 and "urn:schemas-sony-com:service:ScalarWebAPI:1" works for me too (had to resend the query a few times to be sure to get a response). Feels like I get a better response rate now than before, think I only got it to work reliably with "ssdp:all" before, but that is just a feeling...

grolschie

@david thanks. Sometimes with UDP you may need to send 2 packets in quick succession, because delivery is not guaranteed. That's what I'm doing (I wasn't previously). Also I'm now sending on a different thread to listening for the reply. In doing so, now the socket starts listening for a reply before the first packet gets sent. So no replies should be missed.

Upcoming maintenance

HT-ZF9 API and discovery process