HT-ZF9 API and discovery process

TheFC Company

@grolschie
Your tool is working ! Here is a screenshot :
HT-ZF9

I don't know for UTF-8, I'm using this upnp api: https://github.com/custanator/android-upnp-discovery

grolschie

It's not my tool. I wish it was, because then I'd know how to get my Sony AVR to respond to me properly. These Sony's are very picky at responding to M-SEARCH requests I think.

I looked at the the github project, and it's doing something similar to what I am doing and with the same results as me most likely. Maybe it is some Java weirdness? Here is the code in question.

Maybe some Sony person can comment? Perhaps @david please?

david

Unfortunately I have the same problem as you, the devices are very picky then it comes to respond to discovery.
Have had to use Wireshark to compare my messages to another application that worked to get it to work, and compare messages on bit level.
I usually do the same way as @grolschie and use "ssdp:all" since the devices seems more likely to reply to that.
Have also found that the devices in some cases can be very slow to respond so I had to use a longer timeout than expected and some times it take multiple tries.

grolschie

@david Thanks for the reply. Is there any way engineers can update existing products to fix this? Or to get an explanation from them of what magic is actually required to get a response from the device (the official way)?

david

@grolschie I sort of got the normal engineering answer "it works for me".

By the way here are my python script for discovery (haven't runned it for a while and think this version only works with python2). I'm using "ssdp:all" here since I was using it to test DLNA functions. I'm looking for both 'AVTransport', 'ContentDirectory' and 'ScalarWebAPI' functionality in the responce.

#!/usr/bin/env python
# encoding: UTF-8

import socket
import urllib2
import urlparse
import xml.etree.ElementTree as ET
import StringIO

SSDP_BROADCAST_ADDR = '239.255.255.250'
SSDP_BROADCAST_PORT = 1900
SSDP_BROADCAST_PARAMS = [
    'M-SEARCH * HTTP/1.1',
    "HOST: {0}:{1}".format(SSDP_BROADCAST_ADDR, SSDP_BROADCAST_PORT),
    'MAN: "ssdp:discover"', 'MX: 10', 'ST: ssdp:all', '', '']
SSDP_BROADCAST_MSG = '\r\n'.join(SSDP_BROADCAST_PARAMS)

def get_devices(timeout=10.0):
    """Find UPNP devices in current local network"""
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 4)
    s.bind(("", SSDP_BROADCAST_PORT + 10))

    s.sendto(SSDP_BROADCAST_MSG.encode('UTF-8'), (SSDP_BROADCAST_ADDR,
                                                  SSDP_BROADCAST_PORT))

    s.settimeout(timeout)

    devices = []
    while True:
        try:
            data, _addr = s.recvfrom(1024)
        except socket.timeout:
            break

        try:
            info = [a.split(":", 1)
                    for a in data.decode('UTF-8').split('\r\n')[1:]]
            device = dict([(a[0].strip().lower(), a[1].strip())
                           for a in info if len(a) >= 2])
            devices.append(device)
        except Exception:
            pass

    devices_urls = [{'location':dev['location'], 'type':dev['st']}
        for dev in devices if any(x in dev['st']
            for x in ['AVTransport', 'ContentDirectory', 'ScalarWebAPI'])]
    devices = [register_device(location_url) for location_url in devices_urls]

    return devices

def register_device(location_url):
    """Get device information"""
    device_xml_url = location_url['location']

    print "Get device xml from {0}".format(device_xml_url)

    device_xml = urllib2.urlopen(device_xml_url).read().decode('UTF-8')

    info = parse_xml(device_xml)

    location = urlparse.urlparse(device_xml_url)
    hostname = location.hostname

    friendly_name = info.find('./device/friendlyName').text

    if 'ScalarWebAPI' in location_url['type']:
        xpath = './device/X_ScalarWebAPI_DeviceInfo/X_ScalarWebAPI_BaseURL'
        action_url = info.find(xpath).text
    else:
        xpath = "./device/serviceList/service/[serviceType='{0}']/controlURL"
        path = info.find(xpath.format(location_url['type'])).text
        action_url = urlparse.urljoin(location_url['location'], path)

    uuid = info.find('./device/UDN').text

    device = {
        'location': location_url['location'],
        'hostname': hostname,
        'friendly_name': friendly_name,
        'action_url': action_url,
        'st': location_url['type'],
        'uuid':uuid
    }
    return device

def parse_xml(xml_string):
    """Removes namespaces and parse xml"""
    it = ET.iterparse(StringIO.StringIO(xml_string))
    for _, el in it:
        if '}' in el.tag:
            el.tag = el.tag.split('}', 1)[1]  # strip all namespaces
    return it.root

grolschie

Thank you, @david.

TheFC Company

I think I have found the github source for UPnP Tool:
https://github.com/bjtj/upnp-java
(It seems to be the same author)

I will get it a look later

grolschie

@TheFC-Company great find! I will have a more thorough look later. I'm no expert on datagrams, but I'm wondering if it has something to do with joining a multicast group vs merely sending a datagram to a multicast address - the latter is what I am doing and numerous other devices do respond.

Edit: the project also uses a DatagramChannel, which is something I'm unfamiliar with.

TheFC Company

@grolschie
I have sent an email to the UPnP Tool guy, he confirm for the github code, here is it's full reply:

M-SEARCH packet sent from UPnP Tool is like this and you can also check it or more packet logs by clicking "LOG…” Button on the main screen

If you need a java code sending M-SEARCH via datagram socket, please check this code
https://github.com/bjtj/upnp-java/blob/master/src/main/java/com/tjapp/upnp/SSDPMsearchSender.java
Above code is mine and very similar to UPnP Tool’s code

So the code that he use is:

String customQuery = "M-SEARCH * HTTP/1.1\r\n" 
    + "HOST: 239.255.255.250:1900\r\n" 
    + "MAN: \"ssdp:discover\"\r\n"
    + "MX: 3\r\n"
    + "ST: upnp:rootdevice\r\n" 
    + "USER-AGENT: Android/27 UPnP/1.1 UPnPTool/1.5.1"
    + "\r\n";

Unfortunately, that didn't help me to find the device. So we need to dig more in the datagram thing, but I'm not familiar too

grolschie

@TheFC-Company Thanks for the update. I also see that he uses an HttpHeader object rather than just construct a String. Then puts that into a byte array. I wonder why?

grolschie

@TheFC-Company any joy? I have had a further look. So the format of that project's SSDP packet is pretty much the same. The HttpHeader object is just a class in the same project that stores key-value pairs.

Functionally, the only major difference I can see is that he is using a DatagramChannel instead of a DatagramSocket. Hmmm.....

grolschie

I found and fixed a bug in my detection code. I managed to finally get a response from the receiver using the Android emulator, but not from my actual phone. So for now "ssdp:all" shall remain my ST field value.

grolschie

Hi @david. The weirdest thing. Using "ST: urn:schemas-sony.com:service:ScalarWebAPI:1" is now strangely working for me!!!??? I have no idea why! My original search string works (posted back on StackOverflow)!

I'm wondering if the most recent firmware update for the STR-DN1080 changed something. I'm reluctant to roll out the corrected code in case it actually did and some users haven't updated their firmware. The release notes didn't mention anything.

Failing that, a library update at my end or something...

david

@grolschie glad to hear that it is working for you now. Don't know if anything has changed in the last firmware and have limited access to devices right now, but tested on SRS-ZR5 and HT-ZF9 and "urn:schemas-sony-com:service:ScalarWebAPI:1" works for me too (had to resend the query a few times to be sure to get a response). Feels like I get a better response rate now than before, think I only got it to work reliably with "ssdp:all" before, but that is just a feeling...

grolschie

@david thanks. Sometimes with UDP you may need to send 2 packets in quick succession, because delivery is not guaranteed. That's what I'm doing (I wasn't previously). Also I'm now sending on a different thread to listening for the reply. In doing so, now the socket starts listening for a reply before the first packet gets sent. So no replies should be missed.

grolschie

@david You'll never believe this. But today it's not working again! I give up. Back to ssdp:all.
...or maybe not.

Discovery:
Interestingly, if you replace "ssdp" with any word you get even better results than ssdp:all (i.e. less devices responding), e.g. "ST: blah:all\r\n". And the Sony STR-DN1080's DIAL service does respond to this. LOL! This makes processing the results a lot quicker, especially if you have a lot of devices responding to ssdp:all and you are querying every device's device-desc.xml file for the model name. This might help someone.

Edit: actually ST: urn:dial-multiscreen-org:device:dial:1 might be safer.

grolschie

Arrgghh!!! The latest firmware update for the STR-DN1080 has removed the DIAL function from the receiver so my detection routine no longer works! The firmware updates the built-in Chromecast Audio device. Doing some UPnP searches, no DIAL device is detected. And the following file is no longer served by the receiver:
http://ipaddress:8008/ssdp/device-desc.xml

Upcoming maintenance

HT-ZF9 API and discovery process